A little help with child processes.
 

Hi,
I need a little help here managing child processes in Python. I'm not so
skilled in system programming so I hope you can give me some good
suggestions.

I have a very CPU and memory intensive task that has to be repeated many
times (with different input parameters). So I've thought to write a
Python script that create a child process, wait for it to finish and
then starts again another child with different parameters.

The children processes will save all the results to the disk (using
cPickle module), so the parent process has to wait each child only and
then starts the next one. The problem is that I can't work out a
solution. Sorry, but system programming is not my job.

Any suggestions about how to realize that? A skeleton of the script will
just suffice.


Thanks in advance,
Andrea.

Re: A little help with child processes.
 

Unless you are going to start more than one child
process in parallel and there is enough I/O to
make it worthwhile, there's no reason for using
child processes at all. Just program the application
as a single loop. You can't speed up CPU bound
applications with child processes.

HTH,
Larry Bates
Syscon, Inc.
"Rech" <rech@MUORISPAMfastwebnet.it> wrote in message
news:rech-D3F585.16050416062004@individual.net...
> Hi,
> I need a little help here managing child processes in Python. I'm not so
> skilled in system programming so I hope you can give me some good
> suggestions.
>
> I have a very CPU and memory intensive task that has to be repeated many
> times (with different input parameters). So I've thought to write a
> Python script that create a child process, wait for it to finish and
> then starts again another child with different parameters.
>
> The children processes will save all the results to the disk (using
> cPickle module), so the parent process has to wait each child only and
> then starts the next one. The problem is that I can't work out a
> solution. Sorry, but system programming is not my job.
>
> Any suggestions about how to realize that? A skeleton of the script will
> just suffice.
>
>
> Thanks in advance,
> Andrea.

Re: A little help with child processes.
 

Yes I know, but the CPU-intensive tasks eat a lot of memory and for
weird reasons they don't free it (I know bad programming, but re-writing
all the code is impratical at this point). Running them in child
processes solves the problem because when a child dies it frees all the
allocated memory.

Andrea.


In article <voidnXFxAoCxw03d4p2dnA@comcast.com>,
"Larry Bates" <lbates@swamisoft.com> wrote:

> Unless you are going to start more than one child
> process in parallel and there is enough I/O to
> make it worthwhile, there's no reason for using
> child processes at all. Just program the application
> as a single loop. You can't speed up CPU bound
> applications with child processes.
>
> HTH,
> Larry Bates
> Syscon, Inc.