Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Python-list Digest, Vol 112, Issue 79

Thread Tools

Re: Python-list Digest, Vol 112, Issue 79

Niklas Berliner
Posts: n/a
> ---------- Weitergeleitete Nachricht ----------
> From: Dave Angel <(E-Mail Removed)>
> To: Removed)
> Cc:
> Date: Thu, 10 Jan 2013 00:56:20 -0500
> Subject: Re: subprocess.Popen and multiprocessing fails to execute
> external program
> On 01/09/2013 11:08 PM, Niklas Berliner wrote:
> > I have a pipline that involves processing some data, handing the data to

> an
> > external program (t_coffee used for sequence alignments in

> bioinformatics),
> > and postprocessing the result. Since I have a lot of data, I need to run

> my
> > pipeline in parallel which I implemented using the multiprocessing module
> > following Doug Hellmanns blog (
> >

> ).
> >
> > My pipeline works perfectly fine when I run it with the multiprocessing
> > implementation and one consumer, i.e. on one core. If I increase the

> number
> > of consumers, i.e. that multiple instances of my pipeline run in parallel
> > the external program fails with a core dump.
> >

> Could it be that the external program is not designed to have multiple
> simultaneous instances? There are many such programs, some of which
> check for an existing process before allowing another one to get far.
> When using the multiprocessing module, always make sure your externals
> are well-behaved before looking for problems in your multi-code.
> To put it more strongly, a well-written program cannot easily be crashed
> by the parent that launched it.
> --
> DaveA

Hi Dave,

the developers of the external program said that they are using the program
with multiple simultaneous instances. Also, when I execute multiple
simultaneous instances of the external program using a bash wrapper script
on my machine it works (the same wrapper scripts that fail when executed
through python).
Before asking here I have contacted the developers of the external program
but they couldn't help me any further.


Reply With Quote

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: Python-list Digest, Vol 18, Issue 208 Charles Hartman Python 10 03-16-2005 12:37 AM
Re: Python-list Digest, Vol 17, Issue 54 Andrew James Python 2 02-03-2005 03:09 PM
DVD Verdict reviews: TIPTOES, INSTANT BELLY DANCER: A CRASH COURSE IN BELLYDANCE, VOL.1 AND VOL. 2, and more! DVD Verdict DVD Video 0 09-24-2004 09:08 AM
Java Open Source Hub Newsletter - Vol 1, Issue 2 kendzo Java 0 12-05-2003 02:19 PM
Java Open Source Hub Newsletter - Vol 1, Issue 1 kendzo Java 0 11-20-2003 09:52 AM