Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Multiprocessing Pool and functions with many arguments

Reply
Thread Tools

Multiprocessing Pool and functions with many arguments

 
 
psaffrey@googlemail.com
Guest
Posts: n/a
 
      04-29-2009
I'm trying to get to grips with the multiprocessing module, having
only used ParallelPython before.

based on this example:

http://docs.python.org/library/multi...ool-of-workers

what happens if I want my "f" to take more than one argument? I want
to have a list of tuples of arguments and have these correspond the
arguments in f, but it keeps complaining that I only have one argument
(the tuple). Do I have to pass in a tuple and break it up inside f? I
can't use multiple input lists, as I would with regular map.

Thanks,

Peter
 
Reply With Quote
 
 
 
 
Jesse Noller
Guest
Posts: n/a
 
      04-30-2009
On Wed, Apr 29, 2009 at 2:01 PM, http://www.velocityreviews.com/forums/(E-Mail Removed)
<(E-Mail Removed)> wrote:
> I'm trying to get to grips with the multiprocessing module, having
> only used ParallelPython before.
>
> based on this example:
>
> http://docs.python.org/library/multi...ool-of-workers
>
> what happens if I want my "f" to take more than one argument? I want
> to have a list of tuples of arguments and have these correspond the
> arguments in f, but it keeps complaining that I only have one argument
> (the tuple). Do I have to pass in a tuple and break it up inside f? I
> can't use multiple input lists, as I would with regular map.
>
> Thanks,
>
> Peter


Perhaps these articles will help you:

http://www.doughellmann.com/PyMOTW/m....html#pool-map
http://www.doughellmann.com/PyMOTW/m...implemapreduce

jesse
 
Reply With Quote
 
 
 
 
Piet van Oostrum
Guest
Posts: n/a
 
      05-01-2009
>>>>> "(E-Mail Removed)" <(E-Mail Removed)> (P) wrote:

>P> I'm trying to get to grips with the multiprocessing module, having
>P> only used ParallelPython before.


>P> based on this example:


>P> http://docs.python.org/library/multi...ool-of-workers


>P> what happens if I want my "f" to take more than one argument? I want
>P> to have a list of tuples of arguments and have these correspond the
>P> arguments in f, but it keeps complaining that I only have one argument
>P> (the tuple). Do I have to pass in a tuple and break it up inside f? I
>P> can't use multiple input lists, as I would with regular map.


You give the tuple of the arguments for the function:

def f(a, b, c):
return a + b * c

pool = Pool(processes=4) # start 4 worker processes

result = pool.apply(f, (2, 3, 4)) # evaluate "f(2, 3, 4)"

Or if you have a list:

args = [ (2, 3, 4), # arguments for call 1
(5, 6, 7) # arguments for call 2
]

print [pool.apply(f, a) for a in args]

However, as each call to apply wait for its results, this will execute
sequentially instead of parallel.

You can't use map directly as it works only with single argument functions.
>>> print pool.map(f, args)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/pool.py", line 148, in map
return self.map_async(func, iterable, chunksize).get()
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/pool.py", line 422, in get
raise self._value
TypeError: f() takes exactly 3 arguments (1 given)

Is that what you mean?

But you can use a wrapper function:

def wrapf(abc):
return f(*abc)

[later...]

print pool.map(wrapf, args)

This is covered in the examples section of multiprocessing (see
calculate and calculatestar for example).

Or you can use apply_async and later wait for the results:

results = [pool.apply_async(f, a) for a in args]
print [r.get() for r in results]

Now the calls to f are done in parallel, which you can check by putting
a sleep inside f.

--
Piet van Oostrum <(E-Mail Removed)>
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: (E-Mail Removed)
 
Reply With Quote
 
Aaron Brady
Guest
Posts: n/a
 
      05-01-2009
On Apr 29, 1:01*pm, "(E-Mail Removed)"
<(E-Mail Removed)> wrote:
> I'm trying to get to grips with the multiprocessing module, having
> only used ParallelPython before.
>
> based on this example:
>
> http://docs.python.org/library/multi...ng-a-pool-of-w...
>
> what happens if I want my "f" to take more than one argument? I want
> to have a list of tuples of arguments and have these correspond the
> arguments in f, but it keeps complaining that I only have one argument
> (the tuple). Do I have to pass in a tuple and break it up inside f? I
> can't use multiple input lists, as I would with regular map.
>
> Thanks,
>
> Peter


The basic constructor should allow it.

http://docs.python.org/library/multi...-process-class

p = Process(target=f, args=('bob',))

I think you are imagining:

result = pool.apply_async(f, [ *(1,2,3), *(4,5,6) ] )

But I don't know of a way.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
multiprocessing.Pool, its queue, and pre-emption John Ladasky Python 7 09-17-2011 08:34 AM
multiprocessing: Correct usage of pool & queue? Allen Fowler Python 1 09-10-2009 02:37 PM
multiprocessing: pool with blocking queue masher Python 4 07-03-2009 03:56 AM
multiprocessing, pool and process crashes dragonk@gmail.com Python 1 04-30-2009 12:58 PM
functions and arguments.length; passing unknown number of arguments oldyork90 Javascript 10 09-27-2008 03:05 AM



Advertisments