Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Trouble with popen2

Reply
Thread Tools

Trouble with popen2

 
 
Rembrandt Q Einstein
Guest
Posts: n/a
 
      09-27-2004
I am running an external command and I need to know a) when it is done
and b) what it wrote to both stdout and stderr. After a little
searching, I found the popen2 module and used the Popen3 class. I'm
having trouble with it hanging, though.

Here is a very well put (by someone else) posting that describes some
background:

http://mail.python.org/pipermail/pyt...ly/230837.html

I came to a similar conclusion as that poster and his workaround
(independently discovered by me) does do the job he requires. However,
I need to also read stderr, so I made this sample writer and runner:

-------
#!/usr/bin/python

import sys

for line in range(0, int(sys.argv[1])):
print "X" * 120

print >>sys.stderr, "hi"
-------
#!/usr/bin/python

import popen2

f = popen2.Popen3("./writer.py 50000", True)
outs = []
errs = []
while (f.poll() == -1):
errs += f.childerr.readlines()
outs += f.fromchild.readlines()
-----


This hangs in the childerr.readlines(). Is it blocking? If so, why?
In any case, how can I be sure to read ALL data from BOTH stderr and
stdout and not be in danger of hanging?


PS: I just found that if I swap the order of the readlines() statements,
it works. But I don't want to use that until I understand why. I
suspect it's a race condition and I don't want to rely on that.
 
Reply With Quote
 
 
 
 
Rembrandt Q Einstein
Guest
Posts: n/a
 
      09-27-2004
Rembrandt Q Einstein wrote:
> I am running an external command and I need to know a) when it is done
> and b) what it wrote to both stdout and stderr. After a little
> searching, I found the popen2 module and used the Popen3 class. I'm
> having trouble with it hanging, though.
>
> Here is a very well put (by someone else) posting that describes some
> background:
>
> http://mail.python.org/pipermail/pyt...ly/230837.html
>
> I came to a similar conclusion as that poster and his workaround
> (independently discovered by me) does do the job he requires. However,
> I need to also read stderr, so I made this sample writer and runner:
>
> -------
> #!/usr/bin/python
>
> import sys
>
> for line in range(0, int(sys.argv[1])):
> print "X" * 120
>
> print >>sys.stderr, "hi"
> -------
> #!/usr/bin/python
>
> import popen2
>
> f = popen2.Popen3("./writer.py 50000", True)
> outs = []
> errs = []
> while (f.poll() == -1):
> errs += f.childerr.readlines()
> outs += f.fromchild.readlines()
> -----
>
>
> This hangs in the childerr.readlines(). Is it blocking? If so, why? In
> any case, how can I be sure to read ALL data from BOTH stderr and stdout
> and not be in danger of hanging?
>
>
> PS: I just found that if I swap the order of the readlines() statements,
> it works. But I don't want to use that until I understand why. I
> suspect it's a race condition and I don't want to rely on that.


It's possible that when I have child.readlines() first, it consumes all
50000 output lines and then err.readlines() grabs the "hi" and it just
exists. When I have them the other way, err.readlines() blocks (I
thought readlines() was non-blocking, though) and it just hangs.

My real external program isn't nice enough to order the output like
that, so here's a more realistic writer:

-------------
#!/usr/bin/python

import sys
import math

for i in range(0, int(sys.argv[1])):
if math.fmod(i, 100) == 0:
print >>sys.stderr, "hi"
print >>sys.stdout, "X" * 120
----

The challenge is to write a program that will run this one in a
subprocess and read all of both stderr and stdout.
 
Reply With Quote
 
 
 
 
Rembrandt Q Einstein
Guest
Posts: n/a
 
      09-27-2004
Rembrandt Q Einstein wrote:
> Rembrandt Q Einstein wrote:
>
>> I am running an external command and I need to know a) when it is done
>> and b) what it wrote to both stdout and stderr. After a little
>> searching, I found the popen2 module and used the Popen3 class. I'm
>> having trouble with it hanging, though.
>>
>> Here is a very well put (by someone else) posting that describes some
>> background:
>>
>> http://mail.python.org/pipermail/pyt...ly/230837.html
>>
>> I came to a similar conclusion as that poster and his workaround
>> (independently discovered by me) does do the job he requires.
>> However, I need to also read stderr, so I made this sample writer and
>> runner:
>>
>> -------
>> #!/usr/bin/python
>>
>> import sys
>>
>> for line in range(0, int(sys.argv[1])):
>> print "X" * 120
>>
>> print >>sys.stderr, "hi"
>> -------
>> #!/usr/bin/python
>>
>> import popen2
>>
>> f = popen2.Popen3("./writer.py 50000", True)
>> outs = []
>> errs = []
>> while (f.poll() == -1):
>> errs += f.childerr.readlines()
>> outs += f.fromchild.readlines()
>> -----
>>
>>
>> This hangs in the childerr.readlines(). Is it blocking? If so, why?
>> In any case, how can I be sure to read ALL data from BOTH stderr and
>> stdout and not be in danger of hanging?
>>
>>
>> PS: I just found that if I swap the order of the readlines()
>> statements, it works. But I don't want to use that until I understand
>> why. I suspect it's a race condition and I don't want to rely on that.

>
>
> It's possible that when I have child.readlines() first, it consumes all
> 50000 output lines and then err.readlines() grabs the "hi" and it just
> exists. When I have them the other way, err.readlines() blocks (I
> thought readlines() was non-blocking, though) and it just hangs.
>
> My real external program isn't nice enough to order the output like
> that, so here's a more realistic writer:
>
> -------------
> #!/usr/bin/python
>
> import sys
> import math
>
> for i in range(0, int(sys.argv[1])):
> if math.fmod(i, 100) == 0:
> print >>sys.stderr, "hi"
> print >>sys.stdout, "X" * 120
> ----
>
> The challenge is to write a program that will run this one in a
> subprocess and read all of both stderr and stdout.


Ah--despite my new nick, I am an idiot. I should just use select() on
fromchild and childerr. Something like this:

while f.poll() == -1:
select([], [f.fromchild, f.childerr],[], 0)
blah

However, I think I might still miss some data this way.
 
Reply With Quote
 
Albert Hofkamp
Guest
Posts: n/a
 
      09-28-2004
On Mon, 27 Sep 2004 15:27:56 -0400, Rembrandt Q Einstein <hercules.rockefeller@springfield.??.us> wrote:
>> The challenge is to write a program that will run this one in a
>> subprocess and read all of both stderr and stdout.

>
> Ah--despite my new nick, I am an idiot. I should just use select() on
> fromchild and childerr. Something like this:
>
> while f.poll() == -1:
> select([], [f.fromchild, f.childerr],[], 0)
> blah
>
> However, I think I might still miss some data this way.


Nope, this should work, assuming that you catch the output of select,
and act accordingly.
(at least, that is my experience)

Not sure why you need the f.poll() though, I think you can do without.



Maybe an extension to the documentation of popen would be in order, this
post pops up quite often here.

--
Albert
--
Unlike popular belief, the .doc format is not an open publically available format.
 
Reply With Quote
 
P@draigBrady.com
Guest
Posts: n/a
 
      09-29-2004
Rembrandt Q Einstein wrote:
> I am running an external command and I need to know a) when it is done
> and b) what it wrote to both stdout and stderr. After a little
> searching, I found the popen2 module and used the Popen3 class. I'm
> having trouble with it hanging, though.


This should be a FAQ if it's not.
Have a go with: http://www.pixelbeat.org/libs/subProcess.py

Pádraig.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Having trouble converting popen2 to subprocess Daniel Klein Python 3 11-18-2006 02:34 PM
Possible problem with popen2 module A. Lloyd Flanagan Python 2 05-03-2004 02:17 PM
popen2 trouble Diez B. Roggisch Python 2 04-05-2004 12:05 PM
popen2 with large input cherico Python 2 01-29-2004 02:14 PM
popen2 Guy Python 1 08-12-2003 04:57 PM



Advertisments