Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > Finding the first file in a dir, fast, for message queue.

Reply
Thread Tools

Finding the first file in a dir, fast, for message queue.

 
 
Alex Maccaw
Guest
Posts: n/a
 
      01-08-2008
I have a file based msg queue, and msgs are stored five folders down.

When a request for the next msg is received, I need to grab the first
file I can find, as fast as I can.

I've tried Dir.glob, and only selecting the first file. This however is
an awful way of doing it as it loads every file into memory, before
selecting the first.

A bit better is Find.find, which finds files incrementally. However,
this still takes about 0.005 seconds (I presume since it's also
'finding' directories').

Is there a faster way to do this?
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
 
 
 
Robert Klemme
Guest
Posts: n/a
 
      01-08-2008
2008/1/8, Alex Maccaw <(E-Mail Removed)>:
> I have a file based msg queue, and msgs are stored five folders down.
>
> When a request for the next msg is received, I need to grab the first
> file I can find, as fast as I can.
>
> I've tried Dir.glob, and only selecting the first file. This however is
> an awful way of doing it as it loads every file into memory, before
> selecting the first.
>
> A bit better is Find.find, which finds files incrementally. However,
> this still takes about 0.005 seconds (I presume since it's also
> 'finding' directories').
>
> Is there a faster way to do this?


You find 5ms when accessing the file system long? I'd say that's
pretty fast considering what you do (recursive search). I doubt you
will get much improvement as long as you always access the file system
for your search. If you know the change frequency of files then you
could store file system contents in memory and only update every n
seconds / minutes or whatever or have a background thread that
continuously updates your in memory representation.

Kind regards

robert


--
use.inject do |as, often| as.you_can - without end

 
Reply With Quote
 
 
 
 
Alex Maccaw
Guest
Posts: n/a
 
      01-08-2008
> You find 5ms when accessing the file system long? I'd say that's
> pretty fast considering what you do (recursive search). I doubt you
> will get much improvement as long as you always access the file system
> for your search. If you know the change frequency of files then you
> could store file system contents in memory and only update every n
> seconds / minutes or whatever or have a background thread that
> continuously updates your in memory representation.



Well, it's long compared to generating folder names, a guid, and writing
files. It means that I can publish about 1000 msgs per second on to my
queue, but only pull of 243 msgs per second.
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Robert Klemme
Guest
Posts: n/a
 
      01-08-2008
2008/1/8, Alex Maccaw <(E-Mail Removed)>:
> > You find 5ms when accessing the file system long? I'd say that's
> > pretty fast considering what you do (recursive search). I doubt you
> > will get much improvement as long as you always access the file system
> > for your search. If you know the change frequency of files then you
> > could store file system contents in memory and only update every n
> > seconds / minutes or whatever or have a background thread that
> > continuously updates your in memory representation.

>
>
> Well, it's long compared to generating folder names, a guid, and writing
> files. It means that I can publish about 1000 msgs per second on to my
> queue, but only pull of 243 msgs per second.


Here's another option: find all files and put them in a queue. Only
redo search when the queue is empty. This might pay off over all.

robert


--
use.inject do |as, often| as.you_can - without end

 
Reply With Quote
 
Alex Maccaw
Guest
Posts: n/a
 
      01-08-2008
Robert Klemme wrote:
> 2008/1/8, Alex Maccaw <(E-Mail Removed)>:
>> files. It means that I can publish about 1000 msgs per second on to my
>> queue, but only pull of 243 msgs per second.

>
> Here's another option: find all files and put them in a queue. Only
> redo search when the queue is empty. This might pay off over all.
>
> robert


Yes, that's what I've done. Message polling is now the same speed as
publishing (so the overall speed is about 10000 msg per second). If you
interested, here's the queue:

http://code.google.com/p/sparrow
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Open file, get first line, delete first line close file Richard Schneeman Ruby 16 08-26-2008 11:54 PM
Finding Server... Finding Host.... enough already!!! Leesa_Tay@softhome.net Computer Support 2 01-20-2006 10:23 AM
help with my first project on first job, how to read a strange file, thanks a lot!!!!!!! matt Java 9 10-27-2004 03:32 AM
My mind is drawing an absolute blank on breadth first traversal and finding current level Wynan James Java 1 10-06-2003 08:07 PM
Q: (XSLT) finding the position of the first occurrence of an attribute value Jorn W Janneck XML 3 09-05-2003 04:09 AM



Advertisments