Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   Re: select.poll.poll() never blocks (http://www.velocityreviews.com/forums/t670131-re-select-poll-poll-never-blocks.html)

Jean-Paul Calderone 02-12-2009 02:49 AM

Re: select.poll.poll() never blocks
 
On Wed, 11 Feb 2009 18:44:53 -0800 (PST), birdsong <david.birdsong@gmail.com> wrote:
>I'm pretty sure I've exhausted all searches and read all the forums
>Google will turn up related to this issue.
>
>I touch an empty file in a sh shell, fire up the python shell, open
>the file for reading(tried all buffering options), register it with a
>poll object for select.POLLIN and call poll(), but the poll never
>blocks and always returns for the FD, EVENT combination I ask for, but
>the file has not anything written to it.


Filesystem files are always reported as readable and writeable by select,
poll, epoll, etc.

If you want to do non-blocking filesystem I/O, your choices are to use a
native thread (Python's threading module, or another third-party module
which wraps the platform thread API) or to use POSIX AIO (or the mildly
incompatible Linux variant. Of course, AIO has tons of caveats and is
generally in a miserable state, so you probably shouldn't use it. That
leaves you with threads.

Jean-Paul

birdsong 02-12-2009 03:01 AM

Re: select.poll.poll() never blocks
 
So I guess I didn't have a complete understanding of poll, I thought
it returned file descriptors that had registered an event that the
user asked to watch for. In my case, select.POLLIN
Constant Meaning
POLLIN There is data to read

....there ins't any data to be read. The file is readable, yes, but is
that all that the syscall has to offer?


On Feb 11, 6:49*pm, Jean-Paul Calderone <exar...@divmod.com> wrote:
> On Wed, 11 Feb 2009 18:44:53 -0800 (PST), birdsong <david.birds...@gmail.com> wrote:
> >I'm pretty sure I've exhausted all searches and read all the forums
> >Google will turn up related to this issue.

>
> >I touch an empty file in a sh shell, fire up the python shell, open
> >the file for reading(tried all buffering options), register it with a
> >poll object for select.POLLIN and call poll(), but the poll never
> >blocks and always returns for the FD, EVENT combination I ask for, but
> >the file has not anything written to it.

>
> Filesystem files are always reported as readable and writeable by select,
> poll, epoll, etc.
>
> If you want to do non-blocking filesystem I/O, your choices are to use a
> native thread (Python's threading module, or another third-party module
> which wraps the platform thread API) or to use POSIX AIO (or the mildly
> incompatible Linux variant. *Of course, AIO has tons of caveats and is
> generally in a miserable state, so you probably shouldn't use it. *That
> leaves you with threads.
>
> Jean-Paul



Rhodri James 02-12-2009 03:47 AM

Re: select.poll.poll() never blocks
 
On Thu, 12 Feb 2009 03:01:16 -0000, birdsong <david.birdsong@gmail.com>
wrote:

> So I guess I didn't have a complete understanding of poll, I thought
> it returned file descriptors that had registered an event that the
> user asked to watch for. In my case, select.POLLIN
> Constant Meaning
> POLLIN There is data to read
>
> ...there ins't any data to be read. The file is readable, yes, but is
> that all that the syscall has to offer?


To quote the web page referenced in help(select):

"It cannot be used on regular files to determine whether a file has grown
since it was last read."

poll() and friends are wrappers over the socket library "poll" and
"select" functions, which primarily exist to facilitate asynchronous
socket communications. They can take ordinary file descriptors as
well as sockets because there's no good reason not to, but a regular
file will always be ready to read as long as you haven't read past
the end of file.

You're trying to use a regular file as if it was a pipe. My first
obvious question is do you have to do it this way? Do you have
any control over the program writing to file? Can you cause the
output to go to stdout and pipe it into your script's stdin
instead? That would make your life vastly easier.

If you can't, I'm not sure what your best strategy is. I'd
be tempted to use "tail -f filetocheck | yourscript.py" and
palm the job off on an already-written tool. If you want to
do it in Python, the only thing that springs to mind is
periodically checking the size of the file and reading more
when that changes. You'll need to be very careful to keep
what size you think the file is in sync with how much you've
read!

--
Rhodri James *-* Wildebeeste Herder to the Masses

birdsong 02-12-2009 06:09 AM

Re: select.poll.poll() never blocks
 
On Feb 11, 7:47*pm, "Rhodri James" <rho...@wildebst.demon.co.uk>
wrote:
> On Thu, 12 Feb 2009 03:01:16 -0000, birdsong <david.birds...@gmail.com> *
> wrote:
>
> > So I guess I didn't have a complete understanding of poll, I thought
> > it returned file descriptors that had registered an event that the
> > user asked to watch for. *In my case, select.POLLIN
> > Constant * Meaning
> > POLLIN * * There is data to read

>
> > ...there ins't any data to be read. *The file is readable, yes, but is
> > that all that the syscall has to offer?

>
> To quote the web page referenced in help(select):
>
> "It cannot be used on regular files to determine whether a file has grown *
> since it was last read."
>
> poll() and friends are wrappers over the socket library "poll" and
> "select" functions, which primarily exist to facilitate asynchronous
> socket communications. *They can take ordinary file descriptors as
> well as sockets because there's no good reason not to, but a regular
> file will always be ready to read as long as you haven't read past
> the end of file.
>
> You're trying to use a regular file as if it was a pipe. *My first
> obvious question is do you have to do it this way? *Do you have
> any control over the program writing to file? *Can you cause the
> output to go to stdout and pipe it into your script's stdin
> instead? *That would make your life vastly easier.

agreed, this was my original idea, but i figured implementing a tail -
f via poll was alot faster to write and implement. i'm glad at least
know why it wasn't working as expected.

i've already written a tail -f similar to the one found in the link
from a later poster, but i was hoping to use poll to cut down on
unecessary seeks -i'm trying to avoid actions that would drive up
iowait on the boxes this would run on.

thanks for the clarification on the poll family of syscalls.

>
> If you can't, I'm not sure what your best strategy is. *I'd
> be tempted to use "tail -f filetocheck | yourscript.py" and
> palm the job off on an already-written tool. *If you want to
> do it in Python, the only thing that springs to mind is
> periodically checking the size of the file and reading more
> when that changes. *You'll need to be very careful to keep
> what size you think the file is in sync with how much you've
> read!
>
> --
> Rhodri James *-* Wildebeeste Herder to the Masses




All times are GMT. The time now is 10:53 AM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.