Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > subprocess.Popen pipeline bug?

Reply
Thread Tools

subprocess.Popen pipeline bug?

 
 
Marko Rauhamaa
Guest
Posts: n/a
 
      03-13-2008

This tiny program hangs:

================================================== ======================
#!/usr/bin/env python
import subprocess
a = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE,
stdout = subprocess.PIPE)
b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout)
a.stdin.close()
b.wait() # hangs
a.wait() # never reached
================================================== ======================

It shouldn't, should it?

Environment:
================================================== ======================
Python 2.5.1 (r251:54863, Jun 20 2007, 12:14:09)
[GCC 4.1.2 20061115 (prerelease) (SUSE Linux)] on linux2
================================================== ======================


Marko

--
Marko Rauhamaa (E-Mail Removed) http://pacujo.net/marko/
 
Reply With Quote
 
 
 
 
bryanjugglercryptographer@yahoo.com
Guest
Posts: n/a
 
      03-13-2008
Marko Rauhamaa wrote:
> This tiny program hangs:
>
> ================================================== ======================
> #!/usr/bin/env python
> import subprocess
> a = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE,
> stdout = subprocess.PIPE)
> b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout)
> a.stdin.close()
> b.wait() # hangs
> a.wait() # never reached
> ================================================== ======================


To make it work, add close_fds=True in the Popen that creates b.

> It shouldn't, should it?


Not sure. I think what's happening is that the second cat subprocess
never gets EOF on its stdin, because there are still processes with
an open file descriptor for the other end of the pipe.

The Python program closes a.stdin, and let's suppose that's file
descriptor 4. That's not enough, because the subshell that ran cat and
the cat process itself inherited the open file descriptor 4 when they
forked off.

It looks like Popen is smart enough to close the extraneous
descriptors for pipes it created in the same Popen call, but that
one was created in a previous call and passed in.


--
--Bryan
 
Reply With Quote
 
 
 
 
Douglas Wells
Guest
Posts: n/a
 
      03-13-2008
In article <(E-Mail Removed)>,
Marko Rauhamaa <(E-Mail Removed)> writes:
>
> This tiny program hangs:
>
> ================================================== ======================
> #!/usr/bin/env python
> import subprocess
> a = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE,
> stdout = subprocess.PIPE)
> b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout)
> a.stdin.close()
> b.wait() # hangs
> a.wait() # never reached
> ================================================== ======================
>
> It shouldn't, should it?


Yes, it should.

This issue is related to the subtleties of creating a pipeline in
POSIX environments. The problem is that the cat command in
subprocess a never completes because it never encounters an EOF
(on a.stdin). Even though you issue a close call (a.stdin.close ()),
you're not issuing the "last" close. That's because there is still
at least one file descriptor open in subprocess tree b. That
happened because it was open when the subprocess module executed
a POSIX fork call and it got duplicated as part of the fork call.

I don't see any clean and simple way to actually fix this. (That's
one of the reasons why POSIX shells are so complicated.) There
are a couple of work-arounds that you can use:

1) Force close-on-exec on the specific file descriptor:

import subprocess
a = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE,
stdout = subprocess.PIPE)
# ********* beginning of changes
import os, fcntl
fd = a.stdin.fileno ()
old = fcntl.fcntl (fd, fcntl.F_GETFD)
fcntl.fcntl (fd, fcntl.F_SETFD, old | fcntl.FD_CLOEXEC)
# ********* end of changes
b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout)
a.stdin.close()
b.wait()
a.wait()

Or if it happens to not cause undesired side-effects for you, you can
2) Force close-on-exec *all* non-standard file descriptors by using
the close_fds argument to Popen:

import subprocess
a = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE,
stdout = subprocess.PIPE)
# ********* beginning of changes
# b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout)
b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout,
close_fds = True)
# ********* end of changes
a.stdin.close()
b.wait()
a.wait()

Good luck.

- dmw

--
.. Douglas Wells . Connection Technologies .
.. Internet: -sp9804- -at - contek.com- .
 
Reply With Quote
 
Marko Rauhamaa
Guest
Posts: n/a
 
      03-13-2008
http://www.velocityreviews.com/forums/(E-Mail Removed):

> Not sure. I think what's happening is that the second cat subprocess
> never gets EOF on its stdin, because there are still processes with an
> open file descriptor for the other end of the pipe.


You are right. However, the close_fds technique seems a bit
heavy-handed. Well, that's what you get when you try to combine fork and
exec into a single call.


Marko

--
Marko Rauhamaa (E-Mail Removed) http://pacujo.net/marko/
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to describe a pipeline structure in VHDL Ingmar Seifert VHDL 10 11-04-2009 07:25 AM
a pipeline with collision detection skatoulas@hotmail.com VHDL 3 08-02-2005 05:47 PM
New MS Certification in the pipeline? Adam Field Microsoft Certification 0 04-15-2005 08:57 AM
Latches in pipeline design and numeric logic andyesquire@hotmail.com VHDL 3 04-06-2005 11:57 AM
vhdl for data forwarding in a pipeline machine Eqbal Z VHDL 1 10-09-2003 06:25 PM



Advertisments