Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Reading from stdin

Reply
Thread Tools

Reading from stdin

 
 
Luis Zarrabeitia
Guest
Posts: n/a
 
      10-07-2008

I have a problem with this piece of code:

====
import sys
for line in sys.stdin:
print "You said!", line
====

Namely, it seems that the stdin buffers the input, so there is no reply until
a huge amount of text has bin written. The iterator returned by xreadlines
has the same behavior.

The stdin.readline() function doesn't share that behaviour (it returns as soon
as I hit 'enter').

??Is there any way to tell stdin's iterator not to buffer the input? Is it
part of the standard file protocol?

--
Luis Zarrabeitia (aka Kyrie)
Fac. de Matemática y Computación, UH.
http://profesores.matcom.uh.cu/~kyrie
 
Reply With Quote
 
 
 
 
Lawrence D'Oliveiro
Guest
Posts: n/a
 
      10-07-2008
In message <(E-Mail Removed)>, Luis
Zarrabeitia wrote:

> I have a problem with this piece of code:
>
> ====
> import sys
> for line in sys.stdin:
> print "You said!", line
> ====
>
> Namely, it seems that the stdin buffers the input, so there is no reply
> until a huge amount of text has bin written. The iterator returned by
> xreadlines has the same behavior.
>
> The stdin.readline() function doesn't share that behaviour (it returns as
> soon as I hit 'enter').


Perhaps line-buffering simply doesn't apply when you use a file object as an
iterator.
 
Reply With Quote
 
 
 
 
George Sakkis
Guest
Posts: n/a
 
      10-07-2008
Luis Zarrabeitia wrote:
> I have a problem with this piece of code:
>
> ====
> import sys
> for line in sys.stdin:
> print "You said!", line
> ====
>
> Namely, it seems that the stdin buffers the input, so there is no reply until
> a huge amount of text has bin written. The iterator returned by xreadlines
> has the same behavior.
>
> The stdin.readline() function doesn't share that behaviour (it returns as soon
> as I hit 'enter').
>
> ??Is there any way to tell stdin's iterator not to buffer the input? Is it
> part of the standard file protocol?


Not an answer to your actual question, but you can keep the 'for' loop
instead of rewriting it with 'while' using the iter(function,
sentinel) idiom:

for line in iter(sys.stdin.readline, ""):
print "You said!", line

George
 
Reply With Quote
 
Luis Zarrabeitia
Guest
Posts: n/a
 
      10-08-2008
On Tuesday 07 October 2008 05:33:18 pm George Sakkis wrote:
> Not an answer to your actual question, but you can keep the 'for' loop
> instead of rewriting it with 'while' using the iter(function,
> sentinel) idiom:
>
> for line in iter(sys.stdin.readline, ""):
> print "You said!", line


You're right, it's not an answer to my actual question, but I had completely
forgotten about the 'sentinel' idiom. Many thanks... I was trying to do it
with 'itertools', obviously with no luck.

The question still stands (how to turn off the buffering), but this is a nice
workaround until it gets answered.

--
Luis Zarrabeitia (aka Kyrie)
Fac. de Matemática y Computación, UH.
http://profesores.matcom.uh.cu/~kyrie
 
Reply With Quote
 
Luis Zarrabeitia
Guest
Posts: n/a
 
      10-08-2008
On Tuesday 07 October 2008 05:12:28 pm Lawrence D'Oliveiro wrote:
> In message <(E-Mail Removed)>, Luis
>
> Zarrabeitia wrote:
> > I have a problem with this piece of code:
> >
> > ====
> > import sys
> > for line in sys.stdin:
> > print "You said!", line
> > ====
> >
> > Namely, it seems that the stdin buffers the input, so there is no reply
> > until a huge amount of text has bin written. The iterator returned by
> > xreadlines has the same behavior.
> >
> > The stdin.readline() function doesn't share that behaviour (it returns as
> > soon as I hit 'enter').

>
> Perhaps line-buffering simply doesn't apply when you use a file object as
> an iterator.


You cut out the question you replied to, but left the rest. I got a bit
confused until I remembered that *I* wrote the email .

Anyway, I changed the program to:

===
buff = file("test")
for line in buff:
print "you said", line
===

where 'test' is a named pipe (mkfifo test) to see if the line-buffering also
happened with a file object, and it does. As with stdin, nothing gets printed
until the end of the file or it receives a huge amount of lines, but
using '.readline()' works immediately. So it seems that the buffering
behavior happens by default on stdin and file. It makes sense, as type(stdin)
is 'file'. I can't test it now, but I think the sockets also do input
buffering. I guess one doesn't notice it on the general case because disk
reading happens too fast to see the delay.

That raises a related question: is there any use-case where is better to lock
the input until a lot of data is received, even when the requested data is
already available? Output buffering is understandable and desired (how do I
turn it off, by the way?), and even that one wont lock unless requested to
lock (flush), but I can't find examples where input buffering helps.

(full example with pipes)
$ mkfifo test
$ cat > test
[write data here]

on another console, just execute the script.

Oh, I forgot:
Linux 2.6.24, python 2.5.2, Debian's standard build. I don't have windows at
hand to try it.

--
Luis Zarrabeitia (aka Kyrie)
Fac. de Matemática y Computación, UH.
http://profesores.matcom.uh.cu/~kyrie
 
Reply With Quote
 
George Sakkis
Guest
Posts: n/a
 
      10-08-2008
On Oct 7, 8:13*pm, Luis Zarrabeitia <(E-Mail Removed)> wrote:
> On Tuesday 07 October 2008 05:33:18 pm George Sakkis wrote:
>
> > Not an answer to your actual question, but you can keep the 'for' loop
> > instead of rewriting it with 'while' using the iter(function,
> > sentinel) idiom:

>
> > for line in iter(sys.stdin.readline, ""):
> > * * print "You said!", line

>
> You're right, it's not an answer to my actual question, but I had completely
> forgotten about the 'sentinel' idiom. Many thanks... I was trying to do it
> with 'itertools', obviously with no luck.
>
> The question still stands (how to turn off the buffering), but this is a nice
> workaround until it gets answered.


The closest answer I found comes from the docs (http://docs.python.org/
library/stdtypes.html#file-objects):

"""
In order to make a for loop the most efficient way of looping over the
lines of a file (a very common operation), the next() method uses a
hidden read-ahead buffer. As a consequence of using a read-ahead
buffer, combining next() with other file methods (like readline())
does not work right.
"""

I guess the phrasing "hidden read-ahead buffer" implies that buffering
cannot be turned off (or at least it is not intended to even if it's
somehow possible).

George
 
Reply With Quote
 
Luis Zarrabeitia
Guest
Posts: n/a
 
      10-08-2008
On Tuesday 07 October 2008 11:27:19 pm George Sakkis wrote:
> """
> In order to make a for loop the most efficient way of looping over the
> lines of a file (a very common operation), the next() method uses a
> hidden read-ahead buffer. As a consequence of using a read-ahead
> buffer, combining next() with other file methods (like readline())
> does not work right.
> """
>
> I guess the phrasing "hidden read-ahead buffer" implies that buffering
> cannot be turned off (or at least it is not intended to even if it's
> somehow possible).


Hmm. I wonder how those optimizations look like. Apparently, readline() cannot
read from that read-ahead buffer, and that by itself sounds bad. Currently,
if you loop a few times with next, you cannot use readline afterwards until
you seek() to an absolute position.

Actually, I think I may be replying to myself here. I imagine that 'next' will
read a block instead of a character, and look for lines in there, and as the
underlying OS likely blocks until the whole block is read, 'next' cannot
avoid it. That doesn't explain, though, why readline() can't use next's
buffer, why next doesn't have a sensible timeout for interactive sessions
(unless the OS doesn't support it), and why the readahead cannot be turned
off.

I think I'll have to stick for now with the iter(function,sentinel) solution.

And I may try to find next()'s implementation... I guess I'll be downloading
python's source when my bandwidth allows it (or find it on a browseable
repository)

On a related note, help(file.read) shows:

=====
read(...)
read([size]) -> read at most size bytes, returned as a string.

If the size argument is negative or omitted, read until EOF is reached.
Notice that when in non-blocking mode, less data than what was requested
may be returned, even if no size parameter was given.
=====

But it doesn't say how to put the file object in non-blocking mode. (I was
trying to put the file object in non-blocking mode to test next()'s
behavior). ??Ideas?

--
Luis Zarrabeitia (aka Kyrie)
Fac. de Matemática y Computación, UH.
http://profesores.matcom.uh.cu/~kyrie
 
Reply With Quote
 
Gabriel Genellina
Guest
Posts: n/a
 
      10-14-2008
En Wed, 08 Oct 2008 10:28:55 -0300, Luis Zarrabeitia <(E-Mail Removed)>
escribió:

> And I may try to find next()'s implementation... I guess I'll be
> downloading
> python's source when my bandwidth allows it (or find it on a browseable
> repository)


Try http://svn.python.org - in particular,
http://svn.python.org/view/python/tr...s/fileobject.c

--
Gabriel Genellina

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
peek at stdin, flush stdin Johnathan Doe C Programming 5 05-17-2013 04:30 PM
How to pass stdin of a C++ program to the stdin of a process createdwith ShellExecute() Ben C Programming 2 08-29-2009 09:47 PM
STDIN, OUT, ERR and $stdin, out, err - Differences? Terry Cooper Ruby 7 06-09-2009 05:48 AM
Reading from stdin then launching a program that reads from stdin strange behaviour Stefano Sabatini Perl Misc 6 07-29-2007 10:38 PM
Reading stdin once confuses second stdin read Charlie Zender C Programming 6 06-21-2004 01:39 PM



Advertisments