Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > can't use multiprocessing with class factory?

Reply
Thread Tools

can't use multiprocessing with class factory?

 
 
Alan
Guest
Posts: n/a
 
      01-28-2011
Can the below example be fixed to work?
Thanks,
Alan Isaac

import multiprocessing as mp

class Test(object):
pass

def class_factory(x):
class ConcreteTest(Test):
_x = x
return ConcreteTest

def f(cls):
print cls._x

if __name__ == '__main__':
pool = mp.Pool(2)
pool.map(f, [class_factory(i) for i in range(4)])

 
Reply With Quote
 
 
 
 
Robert Kern
Guest
Posts: n/a
 
      01-28-2011
On 1/28/11 1:02 PM, Alan wrote:
> Can the below example be fixed to work?
> Thanks,
> Alan Isaac
>
> import multiprocessing as mp
>
> class Test(object):
> pass
>
> def class_factory(x):
> class ConcreteTest(Test):
> _x = x
> return ConcreteTest
>
> def f(cls):
> print cls._x
>
> if __name__ == '__main__':
> pool = mp.Pool(2)
> pool.map(f, [class_factory(i) for i in range(4)])


Send the (pickleable) factory and the arguments used to construct the instance,
not the unpickleable instance itself.

def g(factory, i):
cls = factory(i)
print cls._x

if __name__ == '__main__':
pool = mp.Pool(2)
pool.map(g, zip([class_factory] * 4, range(4)))

By the way, when asking for help like this, show us what your code did and
describe what results you want. It can often be hard to figure out exactly what
you mean by "work".

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 
Reply With Quote
 
 
 
 
Daniel Urban
Guest
Posts: n/a
 
      01-28-2011
On Fri, Jan 28, 2011 at 20:02, Alan <(E-Mail Removed)> wrote:
> Can the below example be fixed to work?
> Thanks,
> Alan Isaac
>
> import multiprocessing as mp
>
> class Test(object):
> * *pass
>
> def class_factory(x):
> * *class ConcreteTest(Test):
> * * * *_x = x
> * *return ConcreteTest
>
> def f(cls):
> * *print cls._x
>
> if __name__ == '__main__':
> * *pool = mp.Pool(2)
> * *pool.map(f, [class_factory(i) for i in range(4)])


Only classes defined on the top level of a module are picklable (see
http://docs.python.org/dev/py3k/libr...-and-unpickled
). The collections.namedtuple class factory function works around this
limitation by setting the __module__ attribute of the created class,
but I'm not sure if this solution can be used in this case.


Daniel
 
Reply With Quote
 
Hidura
Guest
Posts: n/a
 
      01-28-2011
What is the output?

2011/1/28, Alan <(E-Mail Removed)>:
> Can the below example be fixed to work?
> Thanks,
> Alan Isaac
>
> import multiprocessing as mp
>
> class Test(object):
> pass
>
> def class_factory(x):
> class ConcreteTest(Test):
> _x = x
> return ConcreteTest
>
> def f(cls):
> print cls._x
>
> if __name__ == '__main__':
> pool = mp.Pool(2)
> pool.map(f, [class_factory(i) for i in range(4)])
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>


--
Enviado desde mi dispositivo mvil

Diego I. Hidalgo D.
 
Reply With Quote
 
Robert Kern
Guest
Posts: n/a
 
      01-28-2011
On 1/28/11 1:25 PM, Daniel Urban wrote:

> Only classes defined on the top level of a module are picklable (see
> http://docs.python.org/dev/py3k/libr...-and-unpickled
> ). The collections.namedtuple class factory function works around this
> limitation by setting the __module__ attribute of the created class,
> but I'm not sure if this solution can be used in this case.


namedtuple's trick only works when you assign the created class to a name at the
module level. E.g.

MyFancyTuple = collections.namedtuple(...)

The trick won't work for "anonymous" classes like the above use case.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 
Reply With Quote
 
Alan
Guest
Posts: n/a
 
      01-28-2011
On Jan 28, 2:23 pm, Robert Kern <(E-Mail Removed)> wrote:
> Send the (pickleable) factory and the arguments used to construct the instance,
> not the unpickleable instance itself.
>
> def g(factory, i):
> cls = factory(i)
> print cls._x
>
> if __name__ == '__main__':
> pool = mp.Pool(2)
> pool.map(g, zip([class_factory] * 4, range(4)))




If I change that to g((factory,i)) it does work.

Thanks!

Alan
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Which multiprocessing methods use shared memory? Kevin Ar18 Python 0 07-27-2010 07:12 PM
Which multiprocessing methods use shared memory? Kevin Ar18 Python 0 07-27-2010 06:24 PM
multiprocessing module - could it use google protocol buffers? zLuke Python 0 05-19-2010 11:24 AM
Python 2.6's multiprocessing lock not working on second use? Frdric Sagnes Python 5 01-19-2009 07:40 PM
Nested Class, Member Class, Inner Class, Local Class, Anonymous Class E11 Java 1 10-12-2005 03:34 PM



Advertisments