Re: Python-list Digest, Vol 112, Issue 79
 

>
>
>
>
> ---------- Weitergeleitete Nachricht ----------
> From: Dave Angel <d@davea.name>
> To: python-list@python.org
> Cc:
> Date: Thu, 10 Jan 2013 00:56:20 -0500
> Subject: Re: subprocess.Popen and multiprocessing fails to execute
> external program
> On 01/09/2013 11:08 PM, Niklas Berliner wrote:
> > I have a pipline that involves processing some data, handing the data to
> an
> > external program (t_coffee used for sequence alignments in
> bioinformatics),
> > and postprocessing the result. Since I have a lot of data, I need to run
> my
> > pipeline in parallel which I implemented using the multiprocessing module
> > following Doug Hellmanns blog (
> > http://blog.doughellmann.com/2009/04...ng-part-1.html
> ).
> >
> > My pipeline works perfectly fine when I run it with the multiprocessing
> > implementation and one consumer, i.e. on one core. If I increase the
> number
> > of consumers, i.e. that multiple instances of my pipeline run in parallel
> > the external program fails with a core dump.
> >
>
> Could it be that the external program is not designed to have multiple
> simultaneous instances? There are many such programs, some of which
> check for an existing process before allowing another one to get far.
>
> When using the multiprocessing module, always make sure your externals
> are well-behaved before looking for problems in your multi-code.
>
> To put it more strongly, a well-written program cannot easily be crashed
> by the parent that launched it.
>
>
> --
>
> DaveA
>
>

Hi Dave,

the developers of the external program said that they are using the program
with multiple simultaneous instances. Also, when I execute multiple
simultaneous instances of the external program using a bash wrapper script
on my machine it works (the same wrapper scripts that fail when executed
through python).
Before asking here I have contacted the developers of the external program
but they couldn't help me any further.

Cheers,
Niklas