Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > spawning a process with subprocess

Reply
Thread Tools

spawning a process with subprocess

 
 
bhunter
Guest
Posts: n/a
 
      11-26-2007
Hi,

I've used subprocess with 2.4 several times to execute a process, wait
for it to finish, and then look at its output. Now I want to spawn
the process separately, later check to see if it's finished, and if it
is look at its output. I may want to send a signal at some point to
kill the process. This seems straightforward, but it doesn't seem to
be working.

Here's my test case:

import subprocess, time

cmd = "cat somefile"
thread = subprocess.Popen(args=cmd.split(), shell=True,
stdout=subprocess.PIPE, stdin=subprocess.PIPE,
stderr=subprocess.STDOUT, close_fds=True)

while(1):
time.sleep(1)
if(thread.returncode):
break
else:
print thread.returncode

print "returncode = ", thread.returncode
for line in thread.stdout:
print "stdout:\t",line


This will just print the returncode of None forever until I Ctrl-C it.

Of course, the program works fine if I call thread.communicate(), but
since this waits for the process to finish, that's not what I want.

Any help would be appreciated.

 
Reply With Quote
 
 
 
 
kyosohma@gmail.com
Guest
Posts: n/a
 
      11-26-2007
On Nov 26, 10:54 am, bhunter <(E-Mail Removed)> wrote:
> Hi,
>
> I've used subprocess with 2.4 several times to execute a process, wait
> for it to finish, and then look at its output. Now I want to spawn
> the process separately, later check to see if it's finished, and if it
> is look at its output. I may want to send a signal at some point to
> kill the process. This seems straightforward, but it doesn't seem to
> be working.
>
> Here's my test case:
>
> import subprocess, time
>
> cmd = "cat somefile"
> thread = subprocess.Popen(args=cmd.split(), shell=True,
> stdout=subprocess.PIPE, stdin=subprocess.PIPE,
> stderr=subprocess.STDOUT, close_fds=True)
>
> while(1):
> time.sleep(1)
> if(thread.returncode):
> break
> else:
> print thread.returncode
>
> print "returncode = ", thread.returncode
> for line in thread.stdout:
> print "stdout:\t",line
>
> This will just print the returncode of None forever until I Ctrl-C it.
>
> Of course, the program works fine if I call thread.communicate(), but
> since this waits for the process to finish, that's not what I want.
>
> Any help would be appreciated.


I've read that this sort of thing can be a pain. I'm sure someone will
post and have other views though. I have had some success using
Python's threading module though. There's a pretty good walkthrough
here (it uses wxPython in its example):

http://wiki.wxpython.org/LongRunningTasks

Other places of interest include:

http://aspn.activestate.com/ASPN/Coo.../Recipe/491281
http://uucode.com/texts/pylongopgui/pyguiapp.html
http://sayspy.blogspot.com/2007/11/i...ncurrency.html

If I were doing something like this, I would have the process write
it's output to a file and periodically check to see if the file has
data.

Hopefully someone with more knowledge will come along soon.

Mike
 
Reply With Quote
 
 
 
 
bhunter
Guest
Posts: n/a
 
      11-26-2007
> I've read that this sort of thing can be a pain. I'm sure someone will
> post and have other views though. I have had some success using
> Python's threading module though. There's a pretty good walkthrough
> here (it uses wxPython in its example):
>
> http://wiki.wxpython.org/LongRunningTasks
>
> Other places of interest include:
>
> http://aspn.activestate.com/ASPN/Coo...ncurrency.html
>
> If I were doing something like this, I would have the process write
> it's output to a file and periodically check to see if the file has
> data.
>
> Hopefully someone with more knowledge will come along soon.
>
> Mike


Darn. Is threading the only way to do it? I was hoping not to have
to avoid that. Would have thought that there might be a way for
subprocess to handle this automatically.

Thanks for your help,
Brian
 
Reply With Quote
 
kyosohma@gmail.com
Guest
Posts: n/a
 
      11-26-2007
On Nov 26, 12:27 pm, bhunter <(E-Mail Removed)> wrote:
> > I've read that this sort of thing can be a pain. I'm sure someone will
> > post and have other views though. I have had some success using
> > Python's threading module though. There's a pretty good walkthrough
> > here (it uses wxPython in its example):

>
> >http://wiki.wxpython.org/LongRunningTasks

>
> > Other places of interest include:

>
> >http://aspn.activestate.com/ASPN/Coo.../491281http://...

>
> > If I were doing something like this, I would have the process write
> > it's output to a file and periodically check to see if the file has
> > data.

>
> > Hopefully someone with more knowledge will come along soon.

>
> > Mike

>
> Darn. Is threading the only way to do it? I was hoping not to have
> to avoid that. Would have thought that there might be a way for
> subprocess to handle this automatically.
>
> Thanks for your help,
> Brian


This is just the way I do it...as I said, there are probably some
other people in the group who will have other opinions. By the way,
your statement "I was hoping not to have to avoid that" means that you
hoped to use threading...which I think is contradictory to what you
meant.

Mike
 
Reply With Quote
 
Diez B. Roggisch
Guest
Posts: n/a
 
      11-26-2007
bhunter schrieb:
> Hi,
>
> I've used subprocess with 2.4 several times to execute a process, wait
> for it to finish, and then look at its output. Now I want to spawn
> the process separately, later check to see if it's finished, and if it
> is look at its output. I may want to send a signal at some point to
> kill the process. This seems straightforward, but it doesn't seem to
> be working.
>
> Here's my test case:
>
> import subprocess, time
>
> cmd = "cat somefile"
> thread = subprocess.Popen(args=cmd.split(), shell=True,
> stdout=subprocess.PIPE, stdin=subprocess.PIPE,
> stderr=subprocess.STDOUT, close_fds=True)
>
> while(1):
> time.sleep(1)
> if(thread.returncode):
> break
> else:
> print thread.returncode
>
> print "returncode = ", thread.returncode
> for line in thread.stdout:
> print "stdout:\t",line
>
>
> This will just print the returncode of None forever until I Ctrl-C it.
>
> Of course, the program works fine if I call thread.communicate(), but
> since this waits for the process to finish, that's not what I want.
>
> Any help would be appreciated.


I have difficulties understanding what you are after here. To me it
looks as if everything works as expected. I mean you periodically check
on the liveness of the "thread" - which is what you describe above. All
you are missing IMHO is the actual work in this program.

So

while True:
if do_work():
if thread.returncode:
break
else:
thread.kill()

This assumes that your do_work()-method communicates the wish to end the
sub-process using it's returnvalue.

Diez
 
Reply With Quote
 
bhunter
Guest
Posts: n/a
 
      11-26-2007
>
> This is just the way I do it...as I said, there are probably some
> other people in the group who will have other opinions. By the way,
> your statement "I was hoping not to have to avoid that" means that you
> hoped to use threading...which I think is contradictory to what you
> meant.
>
> Mike


That was a typo. "I was hoping to avoid that" is what it should
read. Proof once again: never type while holding a baby.

Brian
 
Reply With Quote
 
bhunter
Guest
Posts: n/a
 
      11-26-2007
On Nov 26, 1:50 pm, "Diez B. Roggisch" <(E-Mail Removed)> wrote:
> bhunter schrieb:
>
>
>
> > Hi,

>
> > I've used subprocess with 2.4 several times to execute a process, wait
> > for it to finish, and then look at its output. Now I want to spawn
> > the process separately, later check to see if it's finished, and if it
> > is look at its output. I may want to send a signal at some point to
> > kill the process. This seems straightforward, but it doesn't seem to
> > be working.

>
> > Here's my test case:

>
> > import subprocess, time

>
> > cmd = "cat somefile"
> > thread = subprocess.Popen(args=cmd.split(), shell=True,
> > stdout=subprocess.PIPE, stdin=subprocess.PIPE,
> > stderr=subprocess.STDOUT, close_fds=True)

>
> > while(1):
> > time.sleep(1)
> > if(thread.returncode):
> > break
> > else:
> > print thread.returncode

>
> > print "returncode = ", thread.returncode
> > for line in thread.stdout:
> > print "stdout:\t",line

>
> > This will just print the returncode of None forever until I Ctrl-C it.

>
> > Of course, the program works fine if I call thread.communicate(), but
> > since this waits for the process to finish, that's not what I want.

>
> > Any help would be appreciated.

>
> I have difficulties understanding what you are after here. To me it
> looks as if everything works as expected. I mean you periodically check
> on the liveness of the "thread" - which is what you describe above. All
> you are missing IMHO is the actual work in this program.
>
> So
>
> while True:
> if do_work():
> if thread.returncode:
> break
> else:
> thread.kill()
>
> This assumes that your do_work()-method communicates the wish to end the
> sub-process using it's returnvalue.
>
> Diez


If the subprocess had finished, I expect that the returncode will not
be None, and the loop would break. The process hasn't actually
started. I know this because while this simple testcase just cats a
file, the real case submits a simulation job. This job never starts
until after I ctrl-c the program.

Brian
 
Reply With Quote
 
Diez B. Roggisch
Guest
Posts: n/a
 
      11-26-2007
bhunter schrieb:
> On Nov 26, 1:50 pm, "Diez B. Roggisch" <(E-Mail Removed)> wrote:
>> bhunter schrieb:
>>
>>
>>
>>> Hi,
>>> I've used subprocess with 2.4 several times to execute a process, wait
>>> for it to finish, and then look at its output. Now I want to spawn
>>> the process separately, later check to see if it's finished, and if it
>>> is look at its output. I may want to send a signal at some point to
>>> kill the process. This seems straightforward, but it doesn't seem to
>>> be working.
>>> Here's my test case:
>>> import subprocess, time
>>> cmd = "cat somefile"
>>> thread = subprocess.Popen(args=cmd.split(), shell=True,
>>> stdout=subprocess.PIPE, stdin=subprocess.PIPE,
>>> stderr=subprocess.STDOUT, close_fds=True)
>>> while(1):
>>> time.sleep(1)
>>> if(thread.returncode):
>>> break
>>> else:
>>> print thread.returncode
>>> print "returncode = ", thread.returncode
>>> for line in thread.stdout:
>>> print "stdout:\t",line
>>> This will just print the returncode of None forever until I Ctrl-C it.
>>> Of course, the program works fine if I call thread.communicate(), but
>>> since this waits for the process to finish, that's not what I want.
>>> Any help would be appreciated.

>> I have difficulties understanding what you are after here. To me it
>> looks as if everything works as expected. I mean you periodically check
>> on the liveness of the "thread" - which is what you describe above. All
>> you are missing IMHO is the actual work in this program.
>>
>> So
>>
>> while True:
>> if do_work():
>> if thread.returncode:
>> break
>> else:
>> thread.kill()
>>
>> This assumes that your do_work()-method communicates the wish to end the
>> sub-process using it's returnvalue.
>>
>> Diez

>
> If the subprocess had finished, I expect that the returncode will not
> be None, and the loop would break. The process hasn't actually
> started. I know this because while this simple testcase just cats a
> file, the real case submits a simulation job. This job never starts
> until after I ctrl-c the program.


I don't know what the reason is for that, but I've just today worked
with code that exactly uses subprocess as advertised - spawning a
process which runs while the main process occasionally checks inside the
child's logfiles for certain state changes.

What might be though is that you need to consume the subprocesses stdout
in your program - because otherwise it will buffer until a certain
amount (usually 4 or 16k) and then halt.



Diez
 
Reply With Quote
 
bhunter
Guest
Posts: n/a
 
      11-26-2007
On Nov 26, 3:05 pm, "Diez B. Roggisch" <(E-Mail Removed)> wrote:
> bhunter schrieb:
>
>
>
> > On Nov 26, 1:50 pm, "Diez B. Roggisch" <(E-Mail Removed)> wrote:
> >> bhunter schrieb:

>
> >>> Hi,
> >>> I've used subprocess with 2.4 several times to execute a process, wait
> >>> for it to finish, and then look at its output. Now I want to spawn
> >>> the process separately, later check to see if it's finished, and if it
> >>> is look at its output. I may want to send a signal at some point to
> >>> kill the process. This seems straightforward, but it doesn't seem to
> >>> be working.
> >>> Here's my test case:
> >>> import subprocess, time
> >>> cmd = "cat somefile"
> >>> thread = subprocess.Popen(args=cmd.split(), shell=True,
> >>> stdout=subprocess.PIPE, stdin=subprocess.PIPE,
> >>> stderr=subprocess.STDOUT, close_fds=True)
> >>> while(1):
> >>> time.sleep(1)
> >>> if(thread.returncode):
> >>> break
> >>> else:
> >>> print thread.returncode
> >>> print "returncode = ", thread.returncode
> >>> for line in thread.stdout:
> >>> print "stdout:\t",line
> >>> This will just print the returncode of None forever until I Ctrl-C it.
> >>> Of course, the program works fine if I call thread.communicate(), but
> >>> since this waits for the process to finish, that's not what I want.
> >>> Any help would be appreciated.
> >> I have difficulties understanding what you are after here. To me it
> >> looks as if everything works as expected. I mean you periodically check
> >> on the liveness of the "thread" - which is what you describe above. All
> >> you are missing IMHO is the actual work in this program.

>
> >> So

>
> >> while True:
> >> if do_work():
> >> if thread.returncode:
> >> break
> >> else:
> >> thread.kill()

>
> >> This assumes that your do_work()-method communicates the wish to end the
> >> sub-process using it's returnvalue.

>
> >> Diez

>
> > If the subprocess had finished, I expect that the returncode will not
> > be None, and the loop would break. The process hasn't actually
> > started. I know this because while this simple testcase just cats a
> > file, the real case submits a simulation job. This job never starts
> > until after I ctrl-c the program.

>
> I don't know what the reason is for that, but I've just today worked
> with code that exactly uses subprocess as advertised - spawning a
> process which runs while the main process occasionally checks inside the
> child's logfiles for certain state changes.
>
> What might be though is that you need to consume the subprocesses stdout
> in your program - because otherwise it will buffer until a certain
> amount (usually 4 or 16k) and then halt.
>
> Diez


It works? You mean there are place on earth where it really does
this? Excellent!

Still doesn't work for me, though. I first tried changing bufsize to
-1, then 0, then a very large number greater than the number of bytes
in this file (just to be sure). None seemed to have any affect.

Then I modified the code to this:

cmd = "cat /nfs/dv1/bhunter/o52a/verif/gmx/rgx.cc"
args = cmd.split()
thread = subprocess.Popen(args=args, shell=True,
stdout=subprocess.PIPE, stdin=subprocess.PIPE,
stderr=subprocess.STDOUT, close_fds=True, bufsize=300000)

lines = []

try:
while(1):
time.sleep(1)
if(thread.returncode):
break
else:
print thread.returncode
lines.extend(thread.stdout.readlines())
except KeyboardInterrupt:
print lines

print "returncode = ", thread.returncode
for line in thread.stdout:
print "stdout:\t",line


This one hangs after the first print of returncode None. Then, I ctrl-
C it and find that the lines array is empty.

It's my guess that the process is waiting for some input, even though
'cat' clearly does not require anything from stdin. But if I put a
communicate() after Popen and a few other tweaks, then everything
works as expected--but of course not as desired.

Still confused.

Brian
 
Reply With Quote
 
MonkeeSage
Guest
Posts: n/a
 
      11-27-2007
Hi Brian,

Couple of things. You should use poll() on the Popen instance, and
should check it explicitly against None (since a 0 return code,
meaning exit successfully, will be treated as a false condition the
same as None). Also, in your second example, you block the program
when you call readlines on the pipe, since readlines blocks until it
reaches eof (i.e., until pipe closes stdout, i.e., process is
complete). Oh, and you don't have to split the input to the args
option yourself, you can just pass a string. So, putting it all
together, you want something like:

import subprocess, time

cmd = "cat somefile"
proc = subprocess.Popen(args=cmd, shell=True,
stdout=subprocess.PIPE, stdin=subprocess.PIPE,
stderr=subprocess.STDOUT, close_fds=True)

while 1:
time.sleep(1)
if proc.poll() != None:
break
else:
print "waiting on child..."

print "returncode =", proc.returncode

HTH,
Jordan
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Spawning Cmd Window via Subprocess D Python 3 10-18-2009 09:07 PM
subprocess.Popen spawning cmd shells Mrown Python 1 01-09-2008 04:10 PM
spawning a process from webform1 Ashish ASP .Net 0 07-11-2005 10:37 PM
Spawning a process with ASP.NET 2.0 beta 2 Marianne ASP .Net 4 07-07-2005 11:09 AM
Error Spawning Process from ASP.NET hoochiegooch@hotmail.com ASP .Net 4 02-08-2005 04:37 PM



Advertisments