Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Possible problem with popen2 module

Reply
Thread Tools

Possible problem with popen2 module

 
 
A. Lloyd Flanagan
Guest
Posts: n/a
 
      04-30-2004
OK, I've got a weird one I haven't been able to figure out yet.
Admittedly I haven't had time to dig into the library source, but this
behavior certainly doesn't seem right. Here's a test case:

"""Test program to demonstrate problem with popen2 module."""
import popen2

def main (argv):
mycmd = 'python2.3 -c "for i in range(100000):\n print i"'
p = popen2.Popen3(mycmd)
print 'result code is %d' % p.wait()
for line in p.fromchild.xreadlines():
print line,

if __name__ == '__main__':
import sys
main(sys.argv)

As you can see I'm using the popen2.Popen3 class to run a process
which prints a lot of output.

Here's the problem: for small values of the constant in the range()
call, e.g. 1000, this works as expected. For larger values, e.g.
100,000, the program never returns from the p.wait() call after the
child process completes. It appears tbe waiting forever on the
waitpid() call.

This is occuring on a Sun server:
> uname -a

SunOS fred 5.8 Generic_108528-29 sun4u sparc SUNW,Sun-Fire-880

and I've seen the exact same behavior under HP-UX:
> uname -a

HP-UX hpserver B.11.11 U 9000/800 2243344530 unlimited-user license

I don't see this behavior with calling os.popen(). I DO see it with
the current implementation of popen5() from the PEP.

Does anyone know why this is occurring? Is this a bona-fide bug? Or
am I just being stupid somehow?
Thanks!

A. Lloyd Flanagan
Contract Programmer
Richmond, VA
 
Reply With Quote
 
 
 
 
Donn Cave
Guest
Posts: n/a
 
      04-30-2004
In article <(E-Mail Removed) >,
http://www.velocityreviews.com/forums/(E-Mail Removed) (A. Lloyd Flanagan) wrote:
> OK, I've got a weird one I haven't been able to figure out yet.
> Admittedly I haven't had time to dig into the library source, but this
> behavior certainly doesn't seem right. Here's a test case:
>
> """Test program to demonstrate problem with popen2 module."""
> import popen2
>
> def main (argv):
> mycmd = 'python2.3 -c "for i in range(100000):\n print i"'
> p = popen2.Popen3(mycmd)
> print 'result code is %d' % p.wait()
> for line in p.fromchild.xreadlines():
> print line,
>
> if __name__ == '__main__':
> import sys
> main(sys.argv)
>
> As you can see I'm using the popen2.Popen3 class to run a process
> which prints a lot of output.
>
> Here's the problem: for small values of the constant in the range()
> call, e.g. 1000, this works as expected. For larger values, e.g.
> 100,000, the program never returns from the p.wait() call after the
> child process completes. It appears tbe waiting forever on the
> waitpid() call.


> I don't see this behavior with calling os.popen(). I DO see it with
> the current implementation of popen5() from the PEP.
>
> Does anyone know why this is occurring? Is this a bona-fide bug? Or
> am I just being stupid somehow?


Well, I will leave it to you to decide how stupid this was.
Nice job on the problem report though, really covers all the
bases.

Pipes are `slow', fixed size devices. You can write only some
small amount of data to a pipe, and then you have to wait for
the peer process on the other end to read that data and make
more room. Waiting like (and waiting for the peer on reads)
is what makes them slow, which really means interruptible by
signals (just an aside.)

What would work the way you want is a disk file. Redirect
output to a file, wait for the process to exit, and read the
file. Pipes are for processes whose output you want to read
while in progress, and you must do that whether you want to
or not.

You don't have exactly this problem with popen() because you're
not really doing the same thing - it doesn't have a wait(),
just a close(), and close() closes the pipe first, which kills
the process so the wait works.

Donn Cave, (E-Mail Removed)
 
Reply With Quote
 
 
 
 
A. Lloyd Flanagan
Guest
Posts: n/a
 
      05-03-2004
Donn Cave <(E-Mail Removed)> wrote in message news:<(E-Mail Removed)>...
> In article <(E-Mail Removed) >,
> (E-Mail Removed) (A. Lloyd Flanagan) wrote:


> Pipes are `slow', fixed size devices. You can write only some
> small amount of data to a pipe, and then you have to wait for
> the peer process on the other end to read that data and make
> more room. Waiting like (and waiting for the peer on reads)
> is what makes them slow, which really means interruptible by
> signals (just an aside.)
>
> What would work the way you want is a disk file. Redirect


Thanks Donn, it makes sense now. I only feel a little stupid .
I'm certainly learning more about inter-process communcation on this assignment.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Problem with win32pipe.popen2 diego Python 1 01-16-2007 12:02 AM
Problem with select.poll and popen2 Magnus Lycka Python 1 08-30-2005 03:56 PM
popen2 trouble Diez B. Roggisch Python 2 04-05-2004 12:05 PM
popen2 with large input cherico Python 2 01-29-2004 02:14 PM
popen2 Guy Python 1 08-12-2003 04:57 PM



Advertisments