Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Line-by-line processing when stdin is not a tty

Reply
Thread Tools

Line-by-line processing when stdin is not a tty

 
 
RG
Guest
Posts: n/a
 
      08-11-2010
In article <(E-Mail Removed)>,
Cameron Simpson <(E-Mail Removed)> wrote:

> On 11Aug2010 00:11, RG <(E-Mail Removed)> wrote:
> | When stdin is not a tty, Python seems to buffer all the input through
> | EOF before processing any of it:
> |
> | [ron@mickey:~]$ cat | python
> | print 123
> | print 456 <hit ctrl-D here>
> | 123
> | 456
> |
> | Is there a way to get Python to process input line-by-line the way it
> | does when stdin is a TTY even when stdin is not a TTY?
>
> What you're seeing here is not python's behaviour but cat's behaviour.
>
> Almost all programs do line buffering (flush buffer at newline) when the
> file is a terminal (character device) and block buffering (flush when a
> fixed size buffer, typically 8192 bytes or some larger power of 2) when
> the file is not a terminal. This is default behaviour for the stdio
> package.
>
> So "cat" is simply not feeding any data to python until it has a lot of
> it;


I don't think that's right:

[ron@mickey:~]$ cat | cat
123
123
321
321

Cat seems to flush its buffer after every newline. Also:

[ron@mickey:~]$ cat -u | python
print 123
print 456
123
456


> We would need to know
> more about your specific task to suggest workarounds.


I'm writing a system in a different language but want to use a Python
library. I know of lots of ways to do this (embed a Python interpreter,
fire up a python server) but by far the easiest to implement is to have
the main program spawn a Python interpreter and interact with it through
its stdin/stdout. In my code I explicitly force the output stream that
is being piped to Python's stdin to be flushed so I know it's not a
buffering problem on the input side.

rg
 
Reply With Quote
 
 
 
 
RG
Guest
Posts: n/a
 
      08-11-2010
In article <i3ud8e$p9e$02$(E-Mail Removed)-online.com>,
Peter Otten <(E-Mail Removed)> wrote:

> Grant Edwards wrote:
>
> > On 2010-08-11, Tim Harig <(E-Mail Removed)> wrote:
> >> On 2010-08-11, RG <(E-Mail Removed)> wrote:
> >>> When stdin is not a tty, Python seems to buffer all the input through
> >>> EOF before processing any of it:
> >>>
> >>> [ron@mickey:~]$ cat | python
> >>> print 123
> >>> print 456 <hit ctrl-D here>
> >>> 123
> >>> 456
> >>>
> >>> Is there a way to get Python to process input line-by-line the way it
> >>> does when stdin is a TTY even when stdin is not a TTY?
> >>
> >> It would be much better to know the overall purpose of what you are
> >> trying
> >> to achieve. There are may be better ways (ie, sockets) depending what
> >> you
> >> are trying to do. Knowing your target platform would also be helpful.
> >>
> >> For the python interpeter itself, you can can get interactive behavior by
> >> invoking it with the -i option.

> >
> > If you're talking about unbuffered stdin/stdout, the option is -u.
> >
> > I don't really see how the -i option is relevent -- it causes the
> > interpreter to go into interactive mode after running the script.

>
> I'd say the following looks like what the OP was asking for:
>
> $ cat | python -i -c'import sys; sys.ps1=""'
> print sys.stdin.isatty()
> False
> print 1
> 1
> print 2
> 2


That is indeed the behavior I'm looking for.

> (Whether it's useful is yet another question)


It's useful to me I'm trying to access a python library from a
program written in another language for which an equivalent library is
not available. The easiest way to do that is to spawn a Python
interpreter and interact with it through stdin/stdout.

Thanks!

rg
 
Reply With Quote
 
 
 
 
Tim Harig
Guest
Posts: n/a
 
      08-11-2010
On 2010-08-11, RG <(E-Mail Removed)> wrote:
> I'm writing a system in a different language but want to use a Python
> library. I know of lots of ways to do this (embed a Python interpreter,
> fire up a python server) but by far the easiest to implement is to have
> the main program spawn a Python interpreter and interact with it through
> its stdin/stdout.


Or, open python using a socket. That way you have total control over how
the information is transfered, as well as bi-directional transfer.
 
Reply With Quote
 
RG
Guest
Posts: n/a
 
      08-11-2010
In article <i3uo7t$6mk$(E-Mail Removed)>,
Tim Harig <(E-Mail Removed)> wrote:

> On 2010-08-11, RG <(E-Mail Removed)> wrote:
> > I'm writing a system in a different language but want to use a Python
> > library. I know of lots of ways to do this (embed a Python interpreter,
> > fire up a python server) but by far the easiest to implement is to have
> > the main program spawn a Python interpreter and interact with it through
> > its stdin/stdout.

>
> Or, open python using a socket.


You mean a TCP/IP socket? Or a unix domain socket? The former has
security issues, and the latter seems like a lot of work. Or is there
an easy way to do it that I don't know about?

rg
 
Reply With Quote
 
Tim Harig
Guest
Posts: n/a
 
      08-11-2010
On 2010-08-11, RG <(E-Mail Removed)> wrote:
> In article <i3uo7t$6mk$(E-Mail Removed)>,
> Tim Harig <(E-Mail Removed)> wrote:
>
>> On 2010-08-11, RG <(E-Mail Removed)> wrote:
>> > I'm writing a system in a different language but want to use a Python
>> > library. I know of lots of ways to do this (embed a Python interpreter,
>> > fire up a python server) but by far the easiest to implement is to have
>> > the main program spawn a Python interpreter and interact with it through
>> > its stdin/stdout.

>>
>> Or, open python using a socket.

>
> You mean a TCP/IP socket? Or a unix domain socket? The former has
> security issues, and the latter seems like a lot of work. Or is there
> an easy way to do it that I don't know about?


I was referring to unix domain sockets or more specifically stream
pipes. I guess it depends what language you are using and what libraries
you have access to. Under C, working with stream pipes is no more trivial
then using pipe(). You can simply create the socket descriptors using
socketpair(). Keep one of the descriptors for your process and pass the
other to the python child process as both stdin and stdout.
 
Reply With Quote
 
RG
Guest
Posts: n/a
 
      08-11-2010
In article <i3uu74$uga$(E-Mail Removed)>,
Tim Harig <(E-Mail Removed)> wrote:

> On 2010-08-11, RG <(E-Mail Removed)> wrote:
> > In article <i3uo7t$6mk$(E-Mail Removed)>,
> > Tim Harig <(E-Mail Removed)> wrote:
> >
> >> On 2010-08-11, RG <(E-Mail Removed)> wrote:
> >> > I'm writing a system in a different language but want to use a Python
> >> > library. I know of lots of ways to do this (embed a Python interpreter,
> >> > fire up a python server) but by far the easiest to implement is to have
> >> > the main program spawn a Python interpreter and interact with it through
> >> > its stdin/stdout.
> >>
> >> Or, open python using a socket.

> >
> > You mean a TCP/IP socket? Or a unix domain socket? The former has
> > security issues, and the latter seems like a lot of work. Or is there
> > an easy way to do it that I don't know about?

>
> I was referring to unix domain sockets or more specifically stream
> pipes. I guess it depends what language you are using and what libraries
> you have access to. Under C, working with stream pipes is no more trivial
> then using pipe(). You can simply create the socket descriptors using
> socketpair(). Keep one of the descriptors for your process and pass the
> other to the python child process as both stdin and stdout.


Ah. That is in fact exactly what I am doing, and that is how I first
encountered this problem.

rg
 
Reply With Quote
 
RG
Guest
Posts: n/a
 
      08-11-2010
In article <(E-Mail Removed)>,
RG <(E-Mail Removed)> wrote:

> In article <i3uu74$uga$(E-Mail Removed)>,
> Tim Harig <(E-Mail Removed)> wrote:
>
> > On 2010-08-11, RG <(E-Mail Removed)> wrote:
> > > In article <i3uo7t$6mk$(E-Mail Removed)>,
> > > Tim Harig <(E-Mail Removed)> wrote:
> > >
> > >> On 2010-08-11, RG <(E-Mail Removed)> wrote:
> > >> > I'm writing a system in a different language but want to use a Python
> > >> > library. I know of lots of ways to do this (embed a Python
> > >> > interpreter,
> > >> > fire up a python server) but by far the easiest to implement is to
> > >> > have
> > >> > the main program spawn a Python interpreter and interact with it
> > >> > through
> > >> > its stdin/stdout.
> > >>
> > >> Or, open python using a socket.
> > >
> > > You mean a TCP/IP socket? Or a unix domain socket? The former has
> > > security issues, and the latter seems like a lot of work. Or is there
> > > an easy way to do it that I don't know about?

> >
> > I was referring to unix domain sockets or more specifically stream
> > pipes. I guess it depends what language you are using and what libraries
> > you have access to. Under C, working with stream pipes is no more trivial
> > then using pipe(). You can simply create the socket descriptors using
> > socketpair(). Keep one of the descriptors for your process and pass the
> > other to the python child process as both stdin and stdout.

>
> Ah. That is in fact exactly what I am doing, and that is how I first
> encountered this problem.
>
> rg


And now I have encountered another problem:

-> print sys.stdin.encoding
<- None

But when I run from a terminal:

[ron@mickey:~]$ python
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.encoding

'UTF-8'


I thought the value of sys.stdin.encoding was hard-coded into the Python
executable at compile time, but that's obviously wrong. So how does
Python get the value of sys.stdin.encoding?

rg
 
Reply With Quote
 
Cameron Simpson
Guest
Posts: n/a
 
      08-11-2010
On 11Aug2010 12:35, Tim Harig <(E-Mail Removed)> wrote:
| > The buffering is a performance choice. Every write requires a context
| > switch from userspace to kernel space, and availability of data in the
| > pipe will wake up a downstream process blocked trying to read.
| > It is far more efficient to do as few such copies as possible, [...]
|
| Right, I don't question the optimization. I question whether the
| intelligence that performes that optimation should be placed within cat or
| whether it should be placed within the shell. It seems to me that the
| shell has a better idea of how the command is being used and can therefore
| make a better decision about whether or not buffering is appropriate.

I would argue it's not much better placed, though it would be nice if
the control could be issued from there. But it can't.

Regarding the former, in this pipeline:

cat some files... | python filter program | something else

how shall the shell know if the python filter (to take the OP's case)
wants its input line buffered (rare) or block buffered (usually ok)?

What might be useful would be a way to attach an attribute to a pipe
or other file descriptor indicating the desired buffering behaviour
that writers to the file descriptor should adopt.

Of course, the ugly sides to that are how many buffering regimes should
it be possible to express and how and when should the upstream (writing)
program decide to check? In a pipeline the pipes are made _before_ any
of the programs commence because the programs need to be attached to the
pipes (this is done before the programs themselves are dispatched). So,
_after_ dispatch the python-wanting-line-buffering issues an ioctl on
the pipe saying "I want line buffering". However, the upstream program
may well already have commenced operation before that happens. It may
even have run to completion before that happens! So, shall all upstream
programs be required to poll? How often? On every write? Shall they
receive a signal? What if they don't catch it? If the downstream
program _requires_ line buffering then the whole situation is racey
and unreliable.

You can see that on reflection this isn't easy to resolve cleanly from
_outside_ the writing program.

To do it from inside requires all programs to sprout an option like
GNU cat's -u option.

Cheers,
--
Cameron Simpson <(E-Mail Removed)> DoD#743
http://www.cskk.ezoshosting.com/cs/

What progress we are making. In the Middle Ages they would have burned
me. Now they are content with burning my books. - Sigmund Freud
 
Reply With Quote
 
Nobody
Guest
Posts: n/a
 
      08-12-2010
On Wed, 11 Aug 2010 10:32:41 +0000, Tim Harig wrote:

>>> Usually you either
>>> need an option on the upstream program to tell it to line
>>> buffer explicitly

>>
>> once cat had an option -u doing exactly that but nowadays
>> -u seems to be ignored
>>
>> http://www.opengroup.org/onlinepubs/...ities/cat.html

>
> I have to wonder why cat knows or cares.


The issue relates to the standard C library. By convention[1], stdin and
stdout are line-buffered if the descriptor refers to a tty, and are
block-buffered otherwise. stderr is always unbuffered.

Any program which uses stdin and stdout without explicitly setting the
buffering or using fflush() will exhibit this behaviour.

[1] ANSI/ISO C is less specific; C99, 7.19.3p7:

As initially opened, the standard error stream is not fully
buffered; the standard input and standard output streams are
fully buffered if and only if the stream can be determined not
to refer to an interactive device.

POSIX says essentially the same thing:

<http://www.opengroup.org/onlinepubs/9699919799/functions/stdin.html>

 
Reply With Quote
 
RG
Guest
Posts: n/a
 
      08-12-2010
In article <(E-Mail Removed)>,
Nobody <(E-Mail Removed)> wrote:

> On Wed, 11 Aug 2010 10:32:41 +0000, Tim Harig wrote:
>
> >>> Usually you either
> >>> need an option on the upstream program to tell it to line
> >>> buffer explicitly
> >>
> >> once cat had an option -u doing exactly that but nowadays
> >> -u seems to be ignored
> >>
> >> http://www.opengroup.org/onlinepubs/...ities/cat.html

> >
> > I have to wonder why cat knows or cares.

>
> The issue relates to the standard C library. By convention[1], stdin and
> stdout are line-buffered if the descriptor refers to a tty, and are
> block-buffered otherwise. stderr is always unbuffered.
>
> Any program which uses stdin and stdout without explicitly setting the
> buffering or using fflush() will exhibit this behaviour.
>
> [1] ANSI/ISO C is less specific; C99, 7.19.3p7:
>
> As initially opened, the standard error stream is not fully
> buffered; the standard input and standard output streams are
> fully buffered if and only if the stream can be determined not
> to refer to an interactive device.
>
> POSIX says essentially the same thing:
>
> <http://www.opengroup.org/onlinepubs/9699919799/functions/stdin.html>


This doesn't explain why "cat | cat" when run interactively outputs
line-by-line (which it does). STDIN to the first cat is a TTY, but the
second one isn't.

For that matter, you can also do this:

nc -l 1234 | cat

and then telnet localhost 1234 and type at it, and it still works
line-by-line.

rg
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
peek at stdin, flush stdin Johnathan Doe C Programming 5 05-17-2013 04:30 PM
How to pass stdin of a C++ program to the stdin of a process createdwith ShellExecute() Ben C Programming 2 08-29-2009 09:47 PM
STDIN, OUT, ERR and $stdin, out, err - Differences? Terry Cooper Ruby 7 06-09-2009 05:48 AM
WhenProblem calling perldb from a script, the current source line is not properly selected! (keywords: perl, xemacs, debug, gud, terminal, tty) tmstaedt Perl Misc 0 11-12-2007 12:32 PM
Reading stdin once confuses second stdin read Charlie Zender C Programming 6 06-21-2004 01:39 PM



Advertisments