Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > multiprocessing: Correct usage of pool & queue?

Reply
Thread Tools

multiprocessing: Correct usage of pool & queue?

 
 
Allen Fowler
Guest
Posts: n/a
 
      09-04-2009
Hello,

I have a list of tasks/items that I want handed off to
threads/processes to complete. (I would like to stick with process if
I could, since there is some CPU work here. )

Each task involves some calculations and a call to a remote server over urllib2/HTTP.

The
time to complete each task varies from 1 to 20 seconds depending on a
number of factors including variable delay on the remote server.

I would like to:

1) Have a maximum of 20 "in-flight" tasks. (thus worker processes?)

2)
Not overload the external server that each task is calling. No more
than "3 new tasks" per second. More "waiting" tasks may be OK, i need
to test it.

3) Certain tasks in my list must be processed in
the correct order. (I guess the asignment logic must somehow tag those
to by done by the same worker?)


Do any of you have suggestions? Can someone point me in the direction of sample code for this?

Thank you,




 
Reply With Quote
 
 
 
 
Aahz
Guest
Posts: n/a
 
      09-10-2009
In article <(E-Mail Removed)>,
Allen Fowler <(E-Mail Removed)> wrote:
>
>1) Have a maximum of 20 "in-flight" tasks. (thus worker processes?)


Good bet.

>3) Certain tasks in my list must be processed in the correct order. (I
>guess the asignment logic must somehow tag those to by done by the same
>worker?)


The simpler way to do this would be to bundle these tasks into a single
queue object that contains a list of tasks. Each worker iterates over
the list of tasks that it receives, which could be a single task.
--
Aahz ((E-Mail Removed)) <*> http://www.pythoncraft.com/

"To me vi is Zen. To use vi is to practice zen. Every command is a
koan. Profound to the user, unintelligible to the uninitiated. You
discover truth everytime you use it." http://www.velocityreviews.com/forums/(E-Mail Removed)
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Dynamic NAT pool doesn't report full when pool contains interface IP tom Cisco 0 10-09-2009 02:22 AM
does python have a generic object pool like commons-pool in Java Rick Lawson Python 8 07-16-2009 11:25 PM
PIX 501 issue routing between VPN pool and local pool eostrike Cisco 3 10-24-2008 09:43 PM
What should be the correct settings for Recycle Worker Processes in default application pool Jordan Tan ASP .Net 0 05-10-2006 06:20 AM
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. Guoqi Zheng ASP .Net 4 06-03-2004 06:39 PM



Advertisments