Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Monitoring directory for new files on Solaris

Reply
Thread Tools

Monitoring directory for new files on Solaris

 
 
Mikhail Teterin
Guest
Posts: n/a
 
      03-11-2008
Hello!

An application we need to write has to monitor a directory and process any
files appearing in there -- as quickly as possible.

The usual approach to this task is to rescan the directory every once in a
while, which is horrible -- it loads the system keeping the program in RAM
and still has latency measured in seconds...

How can I ask the kernel to notify my program, when the directory is updated
in any way -- via something like poll(), or select(), or

http://developers.sun.com/solaris/ar...ompletion.html

?

Is there a Java way of doing this already (without periodic polling), or
will I need to make my own JNI class?

Thanks for any pointers!

-mi
 
Reply With Quote
 
 
 
 
Wolfgang
Guest
Posts: n/a
 
      03-11-2008
Mikhail Teterin schrieb:
> Hello!
>
> An application we need to write has to monitor a directory and process any
> files appearing in there -- as quickly as possible.
>
> The usual approach to this task is to rescan the directory every once in a
> while, which is horrible -- it loads the system keeping the program in RAM
> and still has latency measured in seconds...


what is a rescan in your sense? to scan an mostly empty dir, why should
that be slow, dont you empty dir after processing? Which language your
app use, pure java?

first idea (not java): install gnu find, scan all 5 seconds for file
newer then 5 sec., but take care (read last sentence).

second idea: in filesystem terms: just look for changes in the
directory, like (again just system commands): "cat /dir > /tmp/dirread1
; sleep 5 ; cat /dir > /tmp/dirread2 ; diff dirread1 dirread2" (sorry,
at the moment I can not proof what cat does on dirs, maybe dd does the
trick).
are there any routines in C or java for just reading the dir themself,
not the inodes in it?

In my opinion: the more difficult part in your software are the file
locking during processing the files and to ashure not to process files
which are still in access of other processes, e.g. which still not
complete or just not closed by the writing process.

W.
 
Reply With Quote
 
 
 
 
Mark Thornton
Guest
Posts: n/a
 
      03-11-2008
Mikhail Teterin wrote:
> Hello!
>
> An application we need to write has to monitor a directory and process any
> files appearing in there -- as quickly as possible.
>
> The usual approach to this task is to rescan the directory every once in a
> while, which is horrible -- it loads the system keeping the program in RAM
> and still has latency measured in seconds...
>
> How can I ask the kernel to notify my program, when the directory is updated
> in any way -- via something like poll(), or select(), or
>
> http://developers.sun.com/solaris/ar...ompletion.html
>
> ?
>
> Is there a Java way of doing this already (without periodic polling), or
> will I need to make my own JNI class?
>
> Thanks for any pointers!
>
> -mi


I seem to remember that on Unix the modification time of the directory
normally changes when there is any change to its content. This may be a
bit quicker than listing the content. While repeated listing the
directory seems ugly, it isn't all that slow or inefficient (even from
Java) as everything is cached. This is especially true if processed
files are removed, thus leaving the directory empty.
Warning, the directory modification time does not change on Windows, so
don't try this approach there.

Mark Thornton
 
Reply With Quote
 
Mark Clements
Guest
Posts: n/a
 
      03-11-2008
On Tue, 11 Mar 2008 18:52:55 -0000, Mikhail Teterin
<(E-Mail Removed)> wrote:

> Hello!
>
> An application we need to write has to monitor a directory and process
> any
> files appearing in there -- as quickly as possible.
>
> The usual approach to this task is to rescan the directory every once in
> a
> while, which is horrible -- it loads the system keeping the program in
> RAM
> and still has latency measured in seconds...
>
> How can I ask the kernel to notify my program, when the directory is
> updated
> in any way -- via something like poll(), or select(), or
>
> http://developers.sun.com/solaris/ar...ompletion.html
>
> ?
>
> Is there a Java way of doing this already (without periodic polling), or
> will I need to make my own JNI class?


from the docs for JNotify:

"JNotify works on both Windows (Windows 2000, XP, Vista) and Linux with
INotify support (Kernel 2.6.14 and above)."

This probably won't help you much if you're on Solaris.

java-fam may give you some ideas.

Mark
 
Reply With Quote
 
Victor Sudakov
Guest
Posts: n/a
 
      03-12-2008
In comp.unix.solaris Mikhail Teterin <(E-Mail Removed)> wrote:

> An application we need to write has to monitor a directory and process any
> files appearing in there -- as quickly as possible.


> The usual approach to this task is to rescan the directory every once in a
> while, which is horrible -- it loads the system keeping the program in RAM
> and still has latency measured in seconds...


> How can I ask the kernel to notify my program, when the directory is updated
> in any way -- via something like poll(), or select(), or


> http://developers.sun.com/solaris/ar...ompletion.html


Is there something like kqueue() in Solaris?
In FreeBSD, many people find the wait_on utility very useful.
See http://freebsd.unixfreunde.de/source..._on-1.1.tar.gz

--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/
 
Reply With Quote
 
Michael Vilain
Guest
Posts: n/a
 
      03-12-2008
In article <fr7r7o$2rg8$(E-Mail Removed)>,
Victor Sudakov <(E-Mail Removed)-spam-here.tomsk.su> wrote:

> In comp.unix.solaris Mikhail Teterin <(E-Mail Removed)> wrote:
>
> > An application we need to write has to monitor a directory and process any
> > files appearing in there -- as quickly as possible.

>
> > The usual approach to this task is to rescan the directory every once in a
> > while, which is horrible -- it loads the system keeping the program in RAM
> > and still has latency measured in seconds...

>
> > How can I ask the kernel to notify my program, when the directory is updated
> > in any way -- via something like poll(), or select(), or

>
> > http://developers.sun.com/solaris/ar...ompletion.html

>
> Is there something like kqueue() in Solaris?
> In FreeBSD, many people find the wait_on utility very useful.
> See http://freebsd.unixfreunde.de/source..._on-1.1.tar.gz


There isn't anything in Solaris that does this but with OpenSolaris, you
could mod the kernel or maybe create a Dtrace script that "wakes up"
when a directory is modified.

--
DeeDee, don't press that button! DeeDee! NO! Dee...



 
Reply With Quote
 
Wolfgang
Guest
Posts: n/a
 
      03-12-2008
Wolfgang schrieb:
>
>
> second idea: in filesystem terms: just look for changes in the
> directory,


I forgot (or in german: den Wald vor lauter Baeumen nicht sehen):
as mentioned in another post, unix has also for directorys as for every
inode the atime/mtime/ctime. I think to monitor just the mtime of the
dir give you the trigger to scan for new files, if you dont have subdirs.

W.
 
Reply With Quote
 
victorfeng1973@yahoo.com
Guest
Posts: n/a
 
      03-12-2008
On Mar 11, 4:38 pm, Wolfgang <(E-Mail Removed)> wrote:
> Mikhail Teterin schrieb:
>
> > Hello!

>
> > An application we need to write has to monitor a directory and process any
> > files appearing in there -- as quickly as possible.

>
> > The usual approach to this task is to rescan the directory every once in a
> > while, which is horrible -- it loads the system keeping the program in RAM
> > and still has latency measured in seconds...

>
> what is a rescan in your sense? to scan an mostly empty dir, why should
> that be slow, dont you empty dir after processing? Which language your
> app use, pure java?
>
> first idea (not java): install gnu find, scan all 5 seconds for file
> newer then 5 sec., but take care (read last sentence).


GNU find has option -cmin, but I could not find option for second.

Victor
 
Reply With Quote
 
Gordon Beaton
Guest
Posts: n/a
 
      03-12-2008
On Tue, 11 Mar 2008 14:52:55 -0400, Mikhail Teterin wrote:
> An application we need to write has to monitor a directory and
> process any files appearing in there -- as quickly as possible.
>
> The usual approach to this task is to rescan the directory every
> once in a while, which is horrible -- it loads the system keeping
> the program in RAM and still has latency measured in seconds...


Three suggestions:

- Poll the last modified time of the directory itself at a suitable
interval. It will be updated when changes are made to the directory
(e.g. files added, removed, renamed). When you see that something
has changed, rescan the directory to see what it was.

- I believe Solaris has an mechanism for subscribing to file system
events. Write a JNI wrapper for that.

- Modify the application producing these files, so that it can notify
your application.

/gordon

--
 
Reply With Quote
 
Gordon Beaton
Guest
Posts: n/a
 
      03-12-2008
On Wed, 12 Mar 2008 08:09:41 -0700 (PDT), http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> GNU find has option -cmin, but I could not find option for second.


Remember what the most recent file was that you processed, and use
"-cnewer" instead. If you have to remove the files as you process
them, you can create a marker file of your own solely for this
purpose.

/gordon

--
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
axis attachment apache temp directory solaris being deleted eunever32@yahoo.co.uk Java 0 03-26-2010 03:18 PM
'unable to execute cc: No such file or directory' with distutilson Solaris Tim Edwards Python 0 07-14-2009 09:11 AM
System.IO.Directory.GetDirectories() and System.IO.Directory.GetFiles() are not returning the specified directory Nathan Sokalski ASP .Net 2 09-06-2007 03:58 PM
Failed to start monitoring changes to <directory?> =?Utf-8?B?ZXRyb3BpYw==?= ASP .Net 0 04-18-2006 04:28 PM
Failed to Start Monitoring Directory Changes =?Utf-8?B?SmVmZiBSZWVk?= ASP .Net 0 02-15-2004 05:51 AM



Advertisments