Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > multiprocessing as batch system

Reply
Thread Tools

multiprocessing as batch system

 
 
Neal Becker
Guest
Posts: n/a
 
      01-21-2010
I'm using multiprocessing as a poor man's batch system. It is working OK,
except that it doesn't seem to do load balancing quite right.

I have a 8-cpu machine. If I start, using multiprocessing pool, calling map
with more than 8 elements, it will start 8 processes. It seems, though,
that when some of them finish, it does not start others right away. The
load average may be a lot less than 8 but still additional processes are not
active.

Is there a way to use multiprocessing to achieve the effect I want? (I
really want a simple batch system, but without complex setup)

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
2.6 multiprocessing and pdb Aaron \Castironpi\ Brady Python 1 10-03-2008 08:05 AM
multiprocessing eats memory Max Ivanov Python 6 09-27-2008 10:29 PM
multiprocessing module (PEP 371) sturlamolden Python 6 06-08-2008 04:33 AM
Parallel/Multiprocessing script design question Amit N Python 4 09-13-2007 08:07 PM
System.Security.SecurityException: Error de solicitud de permiso de tipo System.Net.WebPermission, System, Version=1.0.5000.0, Culture=neutral, PublicKeyToken=b77a5c561934e089. Luis Esteban Valencia ASP .Net 0 07-14-2005 01:43 PM



Advertisments