Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > Too many files open?

Reply
Thread Tools

Too many files open?

 
 
The Weeg
Guest
Posts: n/a
 
      12-16-2004
I wrote a pretty simple web scraping tool, and i'm having it organize
the results into folders based upon search queries. After six hundred
or so of these folders are created, i get a bizarre, "too many files
open" error(Errno::EMFILE).

This is the line of code it crashes on:
`mkdir #{foldername}`

And the only time I ever open and close files in my program is in
these few lines, but the program isn't crashing on these lines:

fileTempFile = File.new(filePath + "full_" + fileName +
".html","w")
fileTempFile.puts(response)
fileLinkText = File.new(filePath + "relevant_" + fileName +
".txt","w")
fileLinkText.puts(link + "\n" +
match.to_s.gsub(/<.*?>/,"").gsub(/&[a-zA-Z0-9#]*?;/,''))
fileLinkText.close
fileTempFile.close


As you can see, the files are being opened and closed almost
immediately.

Is there a different way I should be creating directories rather than
through a shell command?

 
Reply With Quote
 
 
 
 
Eric Hodel
Guest
Posts: n/a
 
      12-16-2004
On 16 Dec 2004, at 10:37, The Weeg wrote:

> I wrote a pretty simple web scraping tool, and i'm having it organize
> the results into folders based upon search queries. After six hundred
> or so of these folders are created, i get a bizarre, "too many files
> open" error(Errno::EMFILE).
>
> This is the line of code it crashes on:
> `mkdir #{foldername}`


This opens a file to recieve the shell's response, instead use
Dir.mkdir or FileUtils.

> And the only time I ever open and close files in my program is in
> these few lines, but the program isn't crashing on these lines:
>
> fileTempFile = File.new(filePath + "full_" + fileName +
> ".html","w")
> fileTempFile.puts(response)
> fileLinkText = File.new(filePath + "relevant_" + fileName +
> ".txt","w")
> fileLinkText.puts(link + "\n" +
> match.to_s.gsub(/<.*?>/,"").gsub(/&[a-zA-Z0-9#]*?;/,''))
> fileLinkText.close
> fileTempFile.close


You should be using blocks here, to ensure the file is really closed.

match = match.to_s
match = match.gsub(/<.*?>/, "")
match = match.gsub(/&[a-zA-Z0-9#]*?;/, "")

link_text = "#{link}\n#{match}"

File.open "#{file_path}full_#{file_name}.html", "w" do |temp_file|
temp_file.puts response
end

File.open "#{file_path}relevant_#{file_name}.txt", "w" do |link_file|
link_file.puts link_text
end

> As you can see, the files are being opened and closed almost
> immediately.


If an exception is raised, your File may not be closed if you don't use
the block form.

> Is there a different way I should be creating directories rather than
> through a shell command?


Dir.mkdir, and FileUtils has (I believe) mkpath as well.



 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
how many hard drives is too many? mattcar Computer Support 6 06-29-2007 01:58 AM
How many warnings is too many? Rhino Java 33 12-22-2005 09:39 PM
How many threads are too many? rbt Python 1 06-11-2005 11:03 PM
How many threads is too many? peelman Java 12 01-15-2005 07:37 AM
Too many (small) vs. too large linked script files in a document... Dag Sunde Javascript 4 12-16-2004 11:38 PM



Advertisments