Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > spawning pyhon apps...

Reply
Thread Tools

spawning pyhon apps...

 
 
bruce
Guest
Posts: n/a
 
      01-09-2009
hi...

toying with an idea.. trying to figure out a good/best way to spawn multiple
python scripts from a parent python app. i'm trying to figure out how to
determine when all child apps have completed, or to possibly determine if
any of the child processes have died/halted..

parent app
spawn child1
spawn child2
spawn child3
.
.
.
spawn childn

do i iterate through a os.waitpid(pid) for each pid of the child processes i
create?

is there another approach? code samples/tutorial...??

i've seen various approaches via google, but not just what i'm looking for..

thanks


 
Reply With Quote
 
 
 
 
Jason Scheirer
Guest
Posts: n/a
 
      01-09-2009
On Jan 9, 2:47*pm, "bruce" <(E-Mail Removed)> wrote:
> hi...
>
> toying with an idea.. trying to figure out a good/best way to spawn multiple
> python scripts from a parent python app. i'm trying to figure out how to
> determine when all child apps have completed, or to possibly determine if
> any of the child processes have died/halted..
>
> parent app
> *spawn child1
> *spawn child2
> *spawn child3
> *.
> *.
> *.
> *spawn childn
>
> do i iterate through a os.waitpid(pid) for each pid of the child processes i
> create?
>
> is there another approach? code samples/tutorial...??
>
> i've seen various approaches via google, but not just what i'm looking for..
>
> thanks


Investigate the subprocess module, you probably want Popen objects.
You can do a poll loop like

my_popenobjects = [subprocess.Popen("foo.py", "--filename=file
%i.txt"%x) for x in xrange(10)]

while any(popenobject.statuscode is None for popenobject in
my_popenobjects):
time.sleep(0.25)

If your tasks are more like function calls and less like shell
scripts, then investigate writing your python as an importable module
and use the multiprocessing module, which will do threading/
subprocessing for you.
 
Reply With Quote
 
 
 
 
Chris Rebert
Guest
Posts: n/a
 
      01-09-2009
On Fri, Jan 9, 2009 at 3:43 PM, bruce <(E-Mail Removed)> wrote:
> hi jason....
>
> forgive me... but in your sample:
> my_popenobjects = [subprocess.Popen("foo.py", "--filename=file
> %i.txt"%x) for x in xrange(10)]
> are you spawning 'foo.py' 10 times? that can't be right!


Indeed, it probably ought to be (note the 2nd pair of brackets):
my_popenobjects = [subprocess.Popen(["foo.py",
"--filename=file%i.txt"%x]) for x in xrange(10)]

> so just what is "foo.py" used for? what am i missing...


It's the name of the program you want to run. In this case, it happens
to be a Python script.

>
> it looks like the my_popenobjects array is iterated through to check the
> statuscode. is the statuscode the value that would be returned from a child
> python script via something like "return(2)"....


It's the POSIX exit code of the program (i.e. what int you return from
main() in a C program, or what you pass to sys.exit() in Python)

> i've seen mention of os.waitpid(..) does this play into waiting for child
> processes to complete, or determine if they've terminated??


Don't know specifically, but that's another, lower-level API. The
`subprocess` module is superior.

Cheers,
Chris

--
Follow the path of the Iguana...
http://rebertia.com
 
Reply With Quote
 
Mark Wooding
Guest
Posts: n/a
 
      01-10-2009
bruce <(E-Mail Removed)> wrote:

> toying with an idea.. trying to figure out a good/best way to spawn
> multiple python scripts from a parent python app. i'm trying to figure
> out how to determine when all child apps have completed, or to
> possibly determine if any of the child processes have died/halted..


You don't say what platform you're using, but you mention os.waitpid so
I'll randomly assume it's Unix-like.

> do i iterate through a os.waitpid(pid) for each pid of the child
> processes i create?


That will technically work, and give you the information you wanted,
though not necessarily in the most timely fashion. It'll block on each
process in turn, waiting for its exit status -- so it'll finish as soon
as all the children are dead, but if (say) the fifth process dies first,
you won't find out until the first four have also passed on.

If you don't have anything better for your program to do, and you're
really on Unix, you can call

kid, status = os.waitpid(0, 0)

to wait for something to happen to any of your process's children; the
kid is the process-id of the child being reported and the status is what
happened to it.

If you do have other things for your process to be doing, then your best
bet is to establish a signal handler for SIGCHLD and installing a
handler of the form

import signal as S
import os as OS
import errno as E

## Children sometimes die. It's sad.
def sigchld(sig, frame):
try:
while True:
kid, status = OS.waitpid(0, OS.WNOHANG)
if kid == 0:
break
## Handle death of KID here.
except OSError, err:
if err.errno != E.ECHILD:
raise

### ...

## Establish handler.
S.signal(S.SIGCHLD, sigchld)

should work.

If you're running on Windows then these tricks won't work. As a grim
hack, you could start a thread per child process and have each thread
wait for its own child (sending the exit status through a queue or
something). I'm afraid I don't know Windows well enough to offer
slicker solutions; maybe someone else can help.

-- [mdw]
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
what is meaning of "@" in pyhon program. Evan Python 10 06-30-2008 10:08 AM
pyhon 2.5.1 build fails Robin Becker Python 0 06-04-2007 11:58 AM
Simple pyhon-based CMS Stewart Midwinter Python 4 04-16-2004 01:12 AM
Spawning thread from ASP.NET (how safe?) Sanjay Kedare ASP .Net 1 11-27-2003 10:32 PM
Pyhon Extensions linker error, unresolved symbols Jim Python 0 11-18-2003 11:13 PM



Advertisments