Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > How to initialize each multithreading Pool worker with an individualvalue?

Reply
Thread Tools

How to initialize each multithreading Pool worker with an individualvalue?

 
 
Valery Khamenya
Guest
Posts: n/a
 
      11-30-2010
Hi,

multithreading.pool Pool has a promissing initializer argument in its
constructor.
However it doesn't look possible to use it to initialize each Pool's
worker with some individual value (I'd wish to be wrong here)

So, how to initialize each multithreading Pool worker with the
individual values?

The typical use case might be a connection pool, say, of 3 workers,
where each of 3 workers has its own TCP/IP port.

from multiprocessing.pool import Pool

def port_initializer(_port):
global port
port = _port

def use_connection(some_packet):
global _port
print "sending data over port # %s" % port

if __name__ == "__main__":
ports=((4001,4002, 4003), )
p = Pool(3, port_initializer, ports) # oops...
some_data_to_send = range(20)
p.map(use_connection, some_data_to_send)


best regards
--
Valery A.Khamenya
 
Reply With Quote
 
 
 
 
Aahz
Guest
Posts: n/a
 
      12-30-2010
In article <(E-Mail Removed)>,
Valery Khamenya <(E-Mail Removed)> wrote:
>
>However it doesn't look possible to use it to initialize each Pool's
>worker with some individual value (I'd wish to be wrong here)
>
>So, how to initialize each multithreading Pool worker with the
>individual values?
>
>The typical use case might be a connection pool, say, of 3 workers,
>where each of 3 workers has its own TCP/IP port.
>
>from multiprocessing.pool import Pool
>
>def port_initializer(_port):
> global port
> port = _port
>
>def use_connection(some_packet):
> global _port
> print "sending data over port # %s" % port
>
>if __name__ == "__main__":
> ports=((4001,4002, 4003), )
> p = Pool(3, port_initializer, ports) # oops...


You probably can't use initargs here. Your port_initializer needs to be
some kind of class instance that works with multiprocessing and emits one
port number when its __call__() method gets invoked. (There may be other
ways to accomplish the same effect, but that's what springs to mind.)
--
Aahz ((E-Mail Removed)) <*> http://www.pythoncraft.com/

"Think of it as evolution in action." --Tony Rand
 
Reply With Quote
 
 
 
 
Adam Tauno Williams
Guest
Posts: n/a
 
      12-30-2010
On Thu, 2010-12-30 at 08:01 -0800, Aahz wrote:
> In article <(E-Mail Removed)>,
> Valery Khamenya <(E-Mail Removed)> wrote:
> >However it doesn't look possible to use it to initialize each Pool's
> >worker with some individual value (I'd wish to be wrong here)
> >So, how to initialize each multithreading Pool worker with the
> >individual values?
> >The typical use case might be a connection pool, say, of 3 workers,
> >where each of 3 workers has its own TCP/IP port.
> >from multiprocessing.pool import Pool
> >def port_initializer(_port):
> > global port
> > port = _port
> >def use_connection(some_packet):
> > global _port
> > print "sending data over port # %s" % port
> >if __name__ == "__main__":
> > ports=((4001,4002, 4003), )
> > p = Pool(3, port_initializer, ports) # oops...

> You probably can't use initargs here. Your port_initializer needs to be
> some kind of class instance that works with multiprocessing and emits one
> port number when its __call__() method gets invoked. (There may be other
> ways to accomplish the same effect, but that's what springs to mind.)


Maybe this is obvious; but it is possible to create a pool of workers
all listening on the same socket. An idle worker will pick-up the
connection. Just open the socket in the initial process and then fork
your workers - they will inherit the file handle and can accept() on it.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
Application pool and Worker process J-T ASP .Net 5 12-05-2005 11:17 PM
Cache Expiration and App Pool Worker Process Recycling Jeremy S. ASP .Net 2 03-28-2005 02:21 PM
Application Pool - Worker Process; Idle Timeout Gordon Smith ASP .Net 1 03-01-2005 09:53 PM
I can map all files (.*) to asp.net worker.How do I map NO FILE to asp.net worker? alex ASP .Net 1 02-04-2005 03:18 AM



Advertisments