Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > multiprocessing as batch system

Thread Tools

multiprocessing as batch system

Neal Becker
Posts: n/a
I'm using multiprocessing as a poor man's batch system. It is working OK,
except that it doesn't seem to do load balancing quite right.

I have a 8-cpu machine. If I start, using multiprocessing pool, calling map
with more than 8 elements, it will start 8 processes. It seems, though,
that when some of them finish, it does not start others right away. The
load average may be a lot less than 8 but still additional processes are not

Is there a way to use multiprocessing to achieve the effect I want? (I
really want a simple batch system, but without complex setup)

Reply With Quote

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
2.6 multiprocessing and pdb Aaron \Castironpi\ Brady Python 1 10-03-2008 08:05 AM
multiprocessing eats memory Max Ivanov Python 6 09-27-2008 10:29 PM
multiprocessing module (PEP 371) sturlamolden Python 6 06-08-2008 04:33 AM
Parallel/Multiprocessing script design question Amit N Python 4 09-13-2007 08:07 PM
System.Security.SecurityException: Error de solicitud de permiso de tipo System.Net.WebPermission, System, Version=1.0.5000.0, Culture=neutral, PublicKeyToken=b77a5c561934e089. Luis Esteban Valencia ASP .Net 0 07-14-2005 01:43 PM