Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > How to get number of bytes written to nonblocking FIFO when EAGAIN is raised?

Reply
Thread Tools

How to get number of bytes written to nonblocking FIFO when EAGAIN is raised?

 
 
Aaron Staley
Guest
Posts: n/a
 
      07-20-2011
Scenario. I have a fifo named 'fifo' on my computer (ubuntu linux)
operating in nonblocking mode for both read and write. Under normal
operation all is good:

Interpreter 1 (writer)
>>> import os
>>> fd = os.open('fifo', os.O_WRONLY | os.O_NONBLOCK)
>>> f = os.fdopen(fd,'wb')
>>> f.write('k')
>>> f.flush()


Interpreter 2 (reader):
>>> import os
>>> fd = os.open('fifo', os.O_NONBLOCK)
>>> f = os.fdopen(fd)
>>> f.read() #after f.flush was done in interpreter 1

'k'

However, if interpreter 1 overfills the FIFO, we get an error (EAGAIN)
>>> f.write('a'*70000)

IOError: [Errno 11] Resource temporarily unavailable

However interpreter 2 still receives data
>> len(f.read())

65536

It looks like interpreter 1 pushed data until the FIFO was full and
then raised the IOError. Interpreter 2 constantly received some, but
not all, of what interpreter 2 tried to send.
Unfortunately, the IOError seems to have no attribute indicating how
much data was successfully sent. I've looked through the docs and
can't seem to figure out how; can anyone land some advice?

Thanks,
Aaron Staley
 
Reply With Quote
 
 
 
 
Roy Smith
Guest
Posts: n/a
 
      07-20-2011
In article
<(E-Mail Removed)>,
Aaron Staley <(E-Mail Removed)> wrote:

> Scenario. I have a fifo named 'fifo' on my computer (ubuntu linux)
> operating in nonblocking mode for both read and write. Under normal
> operation all is good:
>
> Interpreter 1 (writer)
> >>> import os
> >>> fd = os.open('fifo', os.O_WRONLY | os.O_NONBLOCK)
> >>> f = os.fdopen(fd,'wb')

> [...]
> Unfortunately, the IOError seems to have no attribute indicating how
> much data was successfully sent. I've looked through the docs and
> can't seem to figure out how; can anyone land some advice?


I'm thinking you want to skip the os.fdopen() call and use the file
descriptor directly, with os.write(). I've never used this, but it
seems like what you probably want to try.
 
Reply With Quote
 
 
 
 
Adam Skutt
Guest
Posts: n/a
 
      07-20-2011
On Jul 19, 9:19*pm, Aaron Staley <(E-Mail Removed)> wrote:
> However, if interpreter 1 overfills the FIFO, we get an error (EAGAIN)>>>f.write('a'*70000)
>
> IOError: [Errno 11] Resource temporarily unavailable
>
> However interpreter 2 still receives data>> len(f.read())
>
> 65536
>
> It looks like interpreter 1 pushed data until the FIFO was full and
> then raised the IOError. *Interpreter 2 constantly received some, but
> not all, of what interpreter 2 tried to send.
> Unfortunately, the IOError seems to have no attribute indicating how
> much data was successfully sent. *I've looked through the docs and
> can't seem to figure out how; can anyone land some advice?


You need to do as Roy Smith suggested and use the actual OS I/O calls
os.read() and os.write(). os.write() returns the number of bytes
actually written to the underlying descriptor. Python file objects
are akin to FILE structures in C: they perform buffering and other
operations internally that makes them less than suitable for usage
with asynchronous UNIX I/O. In particular, they buffer I/O internally
before writing it to the descriptor, so there's no direct relationship
between calling file.write() and how much data is written to the
stream. In addition, file objects also simply raise the underlying OS
error when it occurs. The UNIX write(2) syscall assumes that you have
been keeping track of how many bytes you've successfully written to
the stream and does not track it for you.

Adam

 
Reply With Quote
 
Aaron Staley
Guest
Posts: n/a
 
      07-21-2011
On Jul 19, 8:15*pm, Adam Skutt <(E-Mail Removed)> wrote:
> On Jul 19, 9:19*pm, Aaron Staley <(E-Mail Removed)> wrote:
>
> > However, if interpreter 1 overfills the FIFO, we get an error (EAGAIN)>>> f.write('a'*70000)

>
> > IOError: [Errno 11] Resource temporarily unavailable

>
> > However interpreter 2 still receives data>> len(f.read())

>
> > 65536

>
> > It looks like interpreter 1 pushed data until the FIFO was full and
> > then raised the IOError. *Interpreter 2 constantly received some, but
> > not all, of what interpreter 2 tried to send.
> > Unfortunately, the IOError seems to have no attribute indicating how
> > much data was successfully sent. *I've looked through the docs and
> > can't seem to figure out how; can anyone land some advice?

>
> You need to do as Roy Smith suggested and use the actual OS I/O calls
> os.read() and os.write(). *os.write() returns the number of bytes
> actually written to the underlying descriptor. *Python file objects
> are akin to FILE structures in C: they perform buffering and other
> operations internally that makes them less than suitable for usage
> with asynchronous UNIX I/O. In particular, they buffer I/O internally
> before writing it to the descriptor, so there's no direct relationship
> between calling file.write() and how much data is written to the
> stream. In addition, file objects also simply raise the underlying OS
> error when it occurs. *The UNIX write(2) syscall assumes that you have
> been keeping track of how many bytes you've successfully written to
> the stream and does not track it for you.
>
> Adam


That's for the info; lower level I/O solved the issue.
That said, is such behavior with the file objects intended? It seems
they don't work correctly with an underlying non-blocking file
descriptor, but the documentation doesn't state such. In fact, it
suggests you can use non-blocking with this comment for file.read:

Also note that when in non-blocking mode, less data than was requested
may be returned, even if no size parameter was given.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
any body having complete code for synchronous fifo or know a link where fifo codes are available plz help chaitu VHDL 1 06-01-2007 07:03 PM
any body having complete code for a synchronous FIFO or know a link where FIFO codes are available chaitu VHDL 1 06-01-2007 03:45 AM
any body having complete code for a synchronous FIFO or know a link where FIFO codes are available chaitu VHDL 1 05-31-2007 03:31 PM
any body having complete code for a synchronous FIFO or know a link where FIFO codes are available chaitu VHDL 0 05-31-2007 02:28 PM



Advertisments