Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > multiprocessing question/error

Reply
Thread Tools

multiprocessing question/error

 
 
Eduardo Lenz
Guest
Posts: n/a
 
      01-17-2009
Hi,

I was using the former processing package with python 2.5 with no problems.
After switching to python 2.6.1 I am having some problems with the same code.
The problem seems to be related to the fact that I am using Pool.map
with a bounded method, since it is inside a class. To clarify a little bit,
let me show some parts of the code ....

class Pygen3(self)....
....
....
....
def calcula(self,indiv):
....
....
....

def evaluate(self):
....
....
indiv = range(mult*self.popsize,(mult+1)*self.popsize)
pool = Pool(processes=nproc)
results = pool.map(self.calcula,indiv)
...
...

the error is the following

Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 522, in __bootstrap_inner
self.run()
File "/usr/lib/python2.6/threading.py", line 477, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
_handle_tasks
put(task)
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
__builtin__.instancemethod failed


Thanks for your help.

--

Eduardo Lenz Cardoso
Dr. Eng.
Associate Professor

State University of Santa Catarina
Department of Mechanical Engineering
89223-100 - Joinville-SC - Brasil

Tel: +55 47 4009-7971 - Fax: +55 47 4009-7940
E-mail: http://www.velocityreviews.com/forums/(E-Mail Removed)
---------------------------------------------


--
Esta mensagem foi verificada pelo sistema de antivírus e
acredita-se estar livre de perigo.

 
Reply With Quote
 
 
 
 
Aaron Brady
Guest
Posts: n/a
 
      01-17-2009
On Jan 16, 11:39*pm, Eduardo Lenz <(E-Mail Removed)> wrote:
> Hi,
>
> I was using the former processing package with python 2.5 with no problems.
> After switching to python 2.6.1 I am having some problems with the same code.
> The problem seems to be related to the fact that I am using Pool.map
> with a bounded method, since it is inside a class. To clarify a little bit,
> let me show some parts of the code ....


Hello,

> class Pygen3(self)....


I infer this was a typo.

> * * ....
> * * ....
> * * ....
> * * def calcula(self,indiv):
> * * * * ....
> * * * * ....
> * * * * ....
>
> * *def evaluate(self):
> * * *....
> * * *....
> * * indiv = range(mult*self.popsize,(mult+1)*self.popsize)
> * *pool = Pool(processes=nproc)
> * *results = pool.map(self.calcula,indiv)


I infer the indentation drifted.

> * *...
> * *...
>
> the error is the following
>
> Exception in thread Thread-1:
> Traceback (most recent call last):
> * File "/usr/lib/python2.6/threading.py", line 522, in __bootstrap_inner
> * * self.run()
> * File "/usr/lib/python2.6/threading.py", line 477, in run
> * * self.__target(*self.__args, **self.__kwargs)
> * File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
> _handle_tasks
> * * put(task)
> PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
> __builtin__.instancemethod failed
>
> Thanks for your help.


Checking on two typos above. I have this working:

from multiprocessing import Process

class C(object):
def m1( self ):
p1 = Process(target=self.m2, args=('bob1',))
p2 = Process(target=self.m2, args=('bob2',))
p1.start()
p2.start()
p1.join()
p2.join()
def m2( self, name ):
print 'hello', name

if __name__ == '__main__':
c= C()
c.m1()

But I get the same error as you with this:

from multiprocessing import Process, Pool

class C(object):
def m1( self ):
arg= range( 10 )
pool= Pool( 5 )
results = pool.map(self.m2,arg)
print results
def m2( self, name ):
print 'hello', name

if __name__ == '__main__':
c= C()
c.m1()

I build a workaround with this:

from multiprocessing import Process, Pool

class C(object):
def m1( self ):
arg= zip( [ self ]* 10, range( 10 ) )
pool= Pool( 5 )
results = pool.map(C_m2,arg)
print results
def m2( self, arg ):
print 'hello', arg
return arg* 2
def C_m2( ar, **kwar ):
return C.m2( *ar, **kwar )

if __name__ == '__main__':
c= C()
c.m1()

Note that you have to explicitly include 'self' as a first parameter
when packing the arguments, and then explicitly unpack it in the
helper function. Also, wrapping 'm2' with '@staticmethod' also fails
with this error:

Exception in thread Thread-1:
Traceback (most recent call last):
File "c:\programs\python26\lib\threading.py", line 522, in
__bootstrap_inner
self.run()
File "c:\programs\python26\lib\threading.py", line 477, in run
self.__target(*self.__args, **self.__kwargs)
File "c:\programs\python26\lib\multiprocessing\pool.py" , line 225,
in _handle_tasks
put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup
__builtin__.function failed

If this works for you, it's possible to make the helper function more
dynamic, by accepting a class name and an attribute name as string
arguments, then retrieving them in the subprocesses.
 
Reply With Quote
 
 
 
 
Gabriel Genellina
Guest
Posts: n/a
 
      01-17-2009
En Sat, 17 Jan 2009 14:14:37 -0200, Eduardo Lenz <(E-Mail Removed)>
escribió:

> Another question: is it possible to
> change a bound method to a function on the fly ? I was wondering if
> something
> like class.__dict__["function_name"] could do the job.


Exactly.
Or, if you already have the method, use its im_func attribute to obtain
the original function. BTW, im_self is the "self" argument.

--
Gabriel Genellina

 
Reply With Quote
 
Eduardo Lenz
Guest
Posts: n/a
 
      01-17-2009
On Saturday 17 January 2009 00:43:35 Aaron Brady wrote:
> On Jan 16, 11:39*pm, Eduardo Lenz <(E-Mail Removed)> wrote:
> > Hi,
> >
> > I was using the former processing package with python 2.5 with no
> > problems. After switching to python 2.6.1 I am having some problems with
> > the same code. The problem seems to be related to the fact that I am
> > using Pool.map with a bounded method, since it is inside a class. To
> > clarify a little bit, let me show some parts of the code ....

>
> Hello,
>
> > class Pygen3(self)....

>
> I infer this was a typo.
>
> > * * ....
> > * * ....
> > * * ....
> > * * def calcula(self,indiv):
> > * * * * ....
> > * * * * ....
> > * * * * ....
> >
> > * *def evaluate(self):
> > * * *....
> > * * *....
> > * * indiv = range(mult*self.popsize,(mult+1)*self.popsize)
> > * *pool = Pool(processes=nproc)
> > * *results = pool.map(self.calcula,indiv)

>
> I infer the indentation drifted.
>
> > * *...
> > * *...
> >
> > the error is the following
> >
> > Exception in thread Thread-1:
> > Traceback (most recent call last):
> > * File "/usr/lib/python2.6/threading.py", line 522, in __bootstrap_inner
> > * * self.run()
> > * File "/usr/lib/python2.6/threading.py", line 477, in run
> > * * self.__target(*self.__args, **self.__kwargs)
> > * File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
> > _handle_tasks
> > * * put(task)
> > PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
> > __builtin__.instancemethod failed
> >
> > Thanks for your help.

>
> Checking on two typos above. I have this working:
>
> from multiprocessing import Process
>
> class C(object):
> def m1( self ):
> p1 = Process(target=self.m2, args=('bob1',))
> p2 = Process(target=self.m2, args=('bob2',))
> p1.start()
> p2.start()
> p1.join()
> p2.join()
> def m2( self, name ):
> print 'hello', name
>
> if __name__ == '__main__':
> c= C()
> c.m1()
>
> But I get the same error as you with this:
>
> from multiprocessing import Process, Pool
>
> class C(object):
> def m1( self ):
> arg= range( 10 )
> pool= Pool( 5 )
> results = pool.map(self.m2,arg)
> print results
> def m2( self, name ):
> print 'hello', name
>
> if __name__ == '__main__':
> c= C()
> c.m1()
>
> I build a workaround with this:
>
> from multiprocessing import Process, Pool
>
> class C(object):
> def m1( self ):
> arg= zip( [ self ]* 10, range( 10 ) )
> pool= Pool( 5 )
> results = pool.map(C_m2,arg)
> print results
> def m2( self, arg ):
> print 'hello', arg
> return arg* 2
> def C_m2( ar, **kwar ):
> return C.m2( *ar, **kwar )
>
> if __name__ == '__main__':
> c= C()
> c.m1()
>
> Note that you have to explicitly include 'self' as a first parameter
> when packing the arguments, and then explicitly unpack it in the
> helper function. Also, wrapping 'm2' with '@staticmethod' also fails
> with this error:
>
> Exception in thread Thread-1:
> Traceback (most recent call last):
> File "c:\programs\python26\lib\threading.py", line 522, in
> __bootstrap_inner
> self.run()
> File "c:\programs\python26\lib\threading.py", line 477, in run
> self.__target(*self.__args, **self.__kwargs)
> File "c:\programs\python26\lib\multiprocessing\pool.py" , line 225,
> in _handle_tasks
> put(task)
> PicklingError: Can't pickle <type 'function'>: attribute lookup
> __builtin__.function failed
>
> If this works for you, it's possible to make the helper function more
> dynamic, by accepting a class name and an attribute name as string
> arguments, then retrieving them in the subprocesses.
> --
> http://mail.python.org/mailman/listinfo/python-list


Thanks for your help ! The only question remaninig is why it iused to work
with the former processing package. Another question: is it possible to
change a bound method to a function on the fly ? I was wondering if something
like class.__dict__["function_name"] could do the job. Actually, I was
wondering with something like a casting, but a guess it is not possible.

Thanks again,
Eduardo.
--

Eduardo Lenz Cardoso
Dr. Eng.
Associate Professor

State University of Santa Catarina
Department of Mechanical Engineering
89223-100 - Joinville-SC - Brasil

Tel: +55 47 4009-7971 - Fax: +55 47 4009-7940
E-mail: (E-Mail Removed)
---------------------------------------------

--
Esta mensagem foi verificada pelo sistema de antivírus e
acredita-se estar livre de perigo.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Using multiprocessing nhwarriors Python 4 10-11-2008 04:48 AM
2.6 multiprocessing and pdb Aaron \Castironpi\ Brady Python 1 10-03-2008 08:05 AM
multiprocessing eats memory Max Ivanov Python 6 09-27-2008 10:29 PM
multiprocessing module (PEP 371) sturlamolden Python 6 06-08-2008 04:33 AM
Parallel/Multiprocessing script design question Amit N Python 4 09-13-2007 08:07 PM



Advertisments