Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > multiprocessing module - isn't it a bug?

Reply
Thread Tools

multiprocessing module - isn't it a bug?

 
 
dmitrey
Guest
Posts: n/a
 
      03-14-2009
# THIS WORKS OK
from multiprocessing import Pool
N = 400
K = 800
processes = 2

def costlyFunction2(z):
r = 0
for k in xrange(1, K+2):
r += z ** (1 / k**1.5)
return r

class ABC:
def __init__(self): pass
def testParallel(self):
po = Pool(processes=processes)
r = po.map(costlyFunction2, xrange(N), chunksize = N/
(10*processes))
A=ABC()
A.testParallel()
print 'done'

# But when I define costlyFunction2 inside of class, it doesn't work:
from multiprocessing import Pool
N = 400
K = 800
processes = 2
class ABC:
def __init__(self): pass
def testParallel(self):
po = Pool(processes=processes)
def costlyFunction2(z):
r = 0
for k in xrange(1, K+2):
r += z ** (1 / k**1.5)
return r
r = po.map(costlyFunction2, xrange(N), chunksize = N/
(10*processes))
A=ABC()
A.testParallel()
print 'done'

Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 522, in
__bootstrap_inner
self.run()
File "/usr/lib/python2.6/threading.py", line 477, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
_handle_tasks
put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup
__builtin__.function failed

This doesn't work for
costlyFunction2 = lambda x: 11
as well; and it doesn't work for imap, apply_async as well (same
error).
So, isn't it a bug, or it can be somehow fixed?
Thank you in advance, D.
 
Reply With Quote
 
 
 
 
Terry Reedy
Guest
Posts: n/a
 
      03-14-2009
dmitrey wrote:
> # THIS WORKS OK
> from multiprocessing import Pool
> N = 400
> K = 800
> processes = 2
>
> def costlyFunction2(z):
> r = 0
> for k in xrange(1, K+2):
> r += z ** (1 / k**1.5)
> return r
>
> class ABC:
> def __init__(self): pass
> def testParallel(self):
> po = Pool(processes=processes)
> r = po.map(costlyFunction2, xrange(N), chunksize = N/
> (10*processes))
> A=ABC()
> A.testParallel()
> print 'done'
>
> # But when I define costlyFunction2 inside of class, it doesn't work:
> from multiprocessing import Pool
> N = 400
> K = 800
> processes = 2
> class ABC:
> def __init__(self): pass
> def testParallel(self):
> po = Pool(processes=processes)
> def costlyFunction2(z):
> r = 0
> for k in xrange(1, K+2):
> r += z ** (1 / k**1.5)
> return r
> r = po.map(costlyFunction2, xrange(N), chunksize = N/
> (10*processes))
> A=ABC()
> A.testParallel()
> print 'done'
>
> Exception in thread Thread-1:
> Traceback (most recent call last):
> File "/usr/lib/python2.6/threading.py", line 522, in
> __bootstrap_inner
> self.run()
> File "/usr/lib/python2.6/threading.py", line 477, in run
> self.__target(*self.__args, **self.__kwargs)
> File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
> _handle_tasks
> put(task)
> PicklingError: Can't pickle <type 'function'>: attribute lookup
> __builtin__.function failed
>
> This doesn't work for
> costlyFunction2 = lambda x: 11
> as well; and it doesn't work for imap, apply_async as well (same
> error).
> So, isn't it a bug, or it can be somehow fixed?


Since the multiproccessing Programming Guidelines say
"Picklability
Ensure that the arguments to the methods of proxies are picklable."
and you got a PicklingError, perhaps 'no' and 'no', but I would research
the pickle module and 'picklability' to be sure.

tjr



 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
multiprocessing module makobu Python 2 01-01-2010 07:26 AM
race/deadlock when creating a multiprocessing.manager instance whileimporting a module ? Sebastien Binet Python 4 12-16-2009 09:57 PM
multiprocessing module and os.close(sys.stdin.fileno()) Graham Dumpleton Python 4 02-22-2009 05:41 AM
Python 2.6, multiprocessing module and BSD YouCanCallMeAl@gmail.com Python 9 10-23-2008 11:34 AM
multiprocessing module (PEP 371) sturlamolden Python 6 06-08-2008 04:33 AM



Advertisments