Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > parallel computations: subprocess.Popen(...).communicate()[0] doesnot work with multiprocessing.Pool

Reply
Thread Tools

parallel computations: subprocess.Popen(...).communicate()[0] doesnot work with multiprocessing.Pool

 
 
Hseu-Ming Chen
Guest
Posts: n/a
 
      06-10-2011
Hi,
I am having an issue when making a shell call from within a
multiprocessing.Process(). Here is the story: i tried to parallelize
the computations in 800-ish Matlab scripts and then save the results
to MySQL. The non-parallel/serial version has been running fine for
about 2 years. However, in the parallel version via multiprocessing
that i'm working on, it appears that the Matlab scripts have never
been kicked off and nothing happened with subprocess.Popen. The debug
printing below does not show up either.

Moreover, even if i replace the Matlab invocation with some trivial
"sed" call, still nothing happens.

Is it possible that the Python interpreter i'm using (version 2.6
released on Oct. 1, 200 is too old? Nevertheless, i would like to
make sure the basic framework i've now is not blatantly wrong.

Below is a skeleton of my Python program:

----------------------------------------------
import subprocess
from multiprocessing import Pool

def worker(DBrow,config):
# run one Matlab script
cmd1 = "/usr/local/bin/matlab ... myMatlab.1.m"
subprocess.Popen([cmd1], shell=True, stdout=subprocess.PIPE).communicate()[0]
print "this does not get printed"

cmd2 = "sed ..."
print subprocess.Popen(cmd2, shell=True,
stdout=subprocess.PIPE).communicate()[0]
print "this does not get printed either"
sys.stdout.flush()

### main program below
.......
# kick off parallel processing
pool = Pool()
for DBrow in DBrows: pool.apply_async(worker,(DBrow,config))
pool.close()
pool.join()
.......
----------------------------------------------

Furthermore, i also tried adding the following:
multiprocessing.current_process().curr_proc.daemon = False
at the beginning of the "worker" function above but to no avail.

Any help would really be appreciated.
 
Reply With Quote
 
 
 
 
Chris Torek
Guest
Posts: n/a
 
      06-12-2011
In article <(E-Mail Removed)>
Hseu-Ming Chen <(E-Mail Removed)> wrote:
>I am having an issue when making a shell call from within a
>multiprocessing.Process(). Here is the story: i tried to parallelize
>the computations in 800-ish Matlab scripts and then save the results
>to MySQL. The non-parallel/serial version has been running fine for
>about 2 years. However, in the parallel version via multiprocessing
>that i'm working on, it appears that the Matlab scripts have never
>been kicked off and nothing happened with subprocess.Popen. The debug
>printing below does not show up either.


I obviously do not have your code, and have not even tried this as
an experiment in a simplified environment, but:

>import subprocess
>from multiprocessing import Pool
>
>def worker(DBrow,config):
> # run one Matlab script
> cmd1 = "/usr/local/bin/matlab ... myMatlab.1.m"
> subprocess.Popen([cmd1], shell=True, stdout=subprocess.PIPE).communicate()[0]
> print "this does not get printed"

...
># kick off parallel processing
>pool = Pool()
>for DBrow in DBrows: pool.apply_async(worker,(DBrow,config))
>pool.close()
>pool.join()


The multiprocessing code makes use of pipes to communicate between
the various subprocesses it creates. I suspect these "extra" pipes
are interfering with your subprocesses, when pool.close() waits
for the Matlab script to do something with its copy of the pipes.
To make the subprocess module close them -- so that Matlab does
not have them in the first place and hence pool.close() cannot get
stuck there -- add "close_fds=True" to the Popen() call.

There could still be issues with competing wait() and/or waitpid()
calls (assuming you are using a Unix-like system, or whatever the
equivalent is for Windows) "eating" the wrong subprocess completion
notifications, but that one is harder to solve in general so
if close_fds fixes things, it was just the pipes. If close_fds
does not fix things, you will probably need to defer the pool.close()
step until after all the subprocesses complete.
--
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (4039.22'N, 11150.29'W) +1 801 277 2603
email: gmail (figure it out) http://web.torek.net/torek/index.html
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
multiple calls to show doesnot work for matplotlib Ravikanth Python 2 07-06-2011 11:31 PM
resizeby in IE7 doesnot work? zhengyan11@gmail.com Javascript 0 02-13-2007 07:25 PM
Webservice Doesnot work. Arthur Dzhelali ASP .Net Web Services 0 10-19-2006 12:38 PM
Microsoft.Web.UI.WebControls doesnot work when upladed to web hosting server dave ASP .Net 3 01-19-2006 03:07 AM
My HTML doesnot work in Netscape Venkat XML 2 09-15-2003 12:13 PM



Advertisments