Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > race/deadlock when creating a multiprocessing.manager instance whileimporting a module ?

Reply
Thread Tools

race/deadlock when creating a multiprocessing.manager instance whileimporting a module ?

 
 
Sebastien Binet
Guest
Posts: n/a
 
      12-14-2009
hi there,

say I have this module

## farnsworth ##
__all__ = [
'mgr',
'maths',
]

from multiprocessing.managers import BaseManager

class MathsClass(object):
def add(self, x, y):
return x + y
def mul(self, x, y):
return x * y

class MyManager(BaseManager):
pass

MyManager.register('Maths', MathsClass)

def _setup():
print "creating a manager..."
mgr = MyManager()
print "starting the manager..."
mgr.start()
print "sciencing faster..."
maths = mgr.Maths()
print maths.add(4,3)
print maths.mul(7,
print "done with sciencing."
return (mgr, maths)

# exec at module import
mgr, maths = _setup()

# prevent hysteresis + clean-up
del _setup
## EOF ##

if I use it like so:

$ python -m farnsworth
creating a manager...
starting the manager...
sciencing faster...
7
56
done with sciencing.

all is fine, but if I try to use it thru an import:
$ python
py> import farnsworth.mgr as mgr
creating a manager...
starting the manager...
sciencing faster...

[stuck for some time... hitting ^C]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "farnsworth.py", line 32, in <module>
mgr, maths = _setup()
File "farnsworth.py", line 25, in _setup
maths = mgr.Maths()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 634, in
temp
token, exp = self._create(typeid, *args, **kwds)
File "/usr/lib/python2.6/multiprocessing/managers.py", line 532, in
_create
conn = self._Client(self._address, authkey=self._authkey)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 140,
in Client
answer_challenge(c, authkey)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 372,
in answer_challenge
message = connection.recv_bytes(256) # reject large
message
KeyboardInterrupt

is this a known limitation/feature of the multiprocessing module ?
is there a workaround (acquiring some import lock maybe) ?

cheers,
sebastien.

PS:
$ python
Python 2.6.4 (r264:75706, Oct 27 2009, 06:25:13)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.

$ python -c 'import sys; print sys.version_info'
(2, 6, 4, 'final', 0)

$ uname -a
Linux farnsworth 2.6.31-ARCH #1 SMP PREEMPT Tue Nov 10 19:01:40 CET
2009 x86_64 Intel(R) Core(TM)2 Duo CPU T9400 @ 2.53GHz GenuineIntel
GNU/Linux
 
Reply With Quote
 
 
 
 
Dave Angel
Guest
Posts: n/a
 
      12-15-2009


Sebastien Binet wrote:
> hi there,
>
> say I have this module
>
> ## farnsworth ##
> __all__ = [
> 'mgr',
> 'maths',
> ]
>
> from multiprocessing.managers import BaseManager
>
> class MathsClass(object):
> def add(self, x, y):
> return x + y
> def mul(self, x, y):
> return x * y
>
> class MyManager(BaseManager):
> pass
>
> MyManager.register('Maths', MathsClass)
>
> def _setup():
> print "creating a manager..."
> mgr = MyManager()
> print "starting the manager..."
> mgr.start()
> print "sciencing faster..."
> maths = mgr.Maths()
> print maths.add(4,3)
> print maths.mul(7,
> print "done with sciencing."
> return (mgr, maths)
>
> # exec at module import
> mgr, maths = _setup()
>
> # prevent hysteresis + clean-up
> del _setup
> ## EOF ##
>
> if I use it like so:
>
> $ python -m farnsworth
> creating a manager...
> starting the manager...
> sciencing faster...
> 7
> 56
> done with sciencing.
>
> all is fine, but if I try to use it thru an import:
> $ python
> py> import farnsworth.mgr as mgr
> creating a manager...
> starting the manager...
> sciencing faster...
>
> [stuck for some time... hitting ^C]
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "farnsworth.py", line 32, in <module>
> mgr, maths = _setup()
> File "farnsworth.py", line 25, in _setup
> maths = mgr.Maths()
> File "/usr/lib/python2.6/multiprocessing/managers.py", line 634, in
> temp
> token, exp = self._create(typeid, *args, **kwds)
> File "/usr/lib/python2.6/multiprocessing/managers.py", line 532, in
> _create
> conn = self._Client(self._address, authkey=self._authkey)
> File "/usr/lib/python2.6/multiprocessing/connection.py", line 140,
> in Client
> answer_challenge(c, authkey)
> File "/usr/lib/python2.6/multiprocessing/connection.py", line 372,
> in answer_challenge
> message = connection.recv_bytes(256) # reject large
> message
> KeyboardInterrupt
>
> is this a known limitation/feature of the multiprocessing module ?
> is there a workaround (acquiring some import lock maybe) ?
>
> cheers,
> sebastien.
>
> PS:
> $ python
> Python 2.6.4 (r264:75706, Oct 27 2009, 06:25:13)
> [GCC 4.4.1] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>
> $ python -c 'import sys; print sys.version_info'
> (2, 6, 4, 'final', 0)
>
> $ uname -a
> Linux farnsworth 2.6.31-ARCH #1 SMP PREEMPT Tue Nov 10 19:01:40 CET
> 2009 x86_64 Intel(R) Core(TM)2 Duo CPU T9400 @ 2.53GHz GenuineIntel
> GNU/Linux
>
>

Since I don't see any other responses, I'll give my guess, even though
I'm not very experienced with the multiprocessing module.

It's my understanding that threads may not be created or destroyed
during an import. So you need to find a way to defer the creation till
the imports are done.

DaveA
 
Reply With Quote
 
 
 
 
Sebastien Binet
Guest
Posts: n/a
 
      12-15-2009
Dave,

[..snip..]
On Tuesday 15 December 2009 02:49:00 Dave Angel wrote:
> Since I don't see any other responses, I'll give my guess, even though
> I'm not very experienced with the multiprocessing module.
>
> It's my understanding that threads may not be created or destroyed
> during an import. So you need to find a way to defer the creation till
> the imports are done.


yes, I figured this much and I already wrapped my module with a ModuleFacade
object which has lazy properties attributes [1].
this solves the problem of import my module, but if the module importing my
module is itself being imported, I am back to square one.

and my module is being used by many people.

I guess I'd need at least a way to detect that somebody is trying to use the
lazy properties of my module facade while an import is "still active".
is there such a mechanism ?

cheers,
sebastien.

[1]
https://svnweb.cern.ch/trac/atlasoff...le/__init__.py
--
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################
 
Reply With Quote
 
pograph
Guest
Posts: n/a
 
      12-16-2009
On Dec 15, 2:57 am, Sebastien Binet <(E-Mail Removed)> wrote:
> Dave,
>
> [..snip..]
> On Tuesday 15 December 2009 02:49:00 Dave Angel wrote:
>
> > Since I don't see any other responses, I'll give my guess, even though
> > I'm not very experienced with the multiprocessing module.

>
> > It's my understanding that threads may not be created or destroyed
> > during an import. So you need to find a way to defer the creation till
> > the imports are done.

>
> yes, I figured this much and I already wrapped my module with a ModuleFacade
> object which has lazy properties attributes [1].
> this solves the problem of import my module, but if the module importing my
> module is itself being imported, I am back to square one.
>
> and my module is being used by many people.
>
> I guess I'd need at least a way to detect that somebody is trying to use the
> lazy properties of my module facade while an import is "still active".
> is there such a mechanism ?
>
> cheers,
> sebastien.
>
> [1]https://svnweb.cern.ch/trac/atlasoff/browser/Tools/PyUtils/trunk/pyth...
> --
> #########################################
> # Dr. Sebastien Binet
> # Laboratoire de l'Accelerateur Lineaire
> # Universite Paris-Sud XI
> # Batiment 200
> # 91898 Orsay
> #########################################


The document of multiprocessing says:
Functionality within this package requires that the __main__ method be
importable by the children. This is covered in Programming guidelines
however it is worth pointing out here. This means that some examples,
such as the multiprocessing.Pool examples will not work in the
interactive interpreter.
 
Reply With Quote
 
Sebastien Binet
Guest
Posts: n/a
 
      12-16-2009
hi there,

> On Tuesday 15 December 2009 02:49:00 Dave Angel wrote:
> > Since I don't see any other responses, I'll give my guess, even though
> > I'm not very experienced with the multiprocessing module.
> >
> > It's my understanding that threads may not be created or destroyed
> > during an import. So you need to find a way to defer the creation till
> > the imports are done.

>
> yes, I figured this much and I already wrapped my module with a
> ModuleFacade object which has lazy properties attributes [1].
> this solves the problem of import my module, but if the module importing my
> module is itself being imported, I am back to square one.
>
> and my module is being used by many people.
>
> I guess I'd need at least a way to detect that somebody is trying to use
> the lazy properties of my module facade while an import is "still active".
> is there such a mechanism ?

digging thru the imp.py module, I devised this simple snippet:
import imp
[...]
if imp.lock_held():
raise RuntimeError('some crystal clear error message')

thanks for the help

cheers,
sebastien.
--
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Creating an instance when the argument is already an instance. Olive Python 4 07-06-2012 02:01 PM
Creating class instance from module and class name gentlestone Python 6 10-06-2009 08:13 AM
Re: module docstring, documentation,anything? please note is the module type/object NOT some module Maric Michaud Python 0 06-24-2006 12:42 PM
Module nesting and module vs. instance methods gga Ruby 3 02-19-2005 10:50 PM
Cannot refer to an instance member of a class from within a shared method or shared member initializer without an explicit instance of the class. DJ Dev ASP .Net 3 02-08-2004 04:19 PM



Advertisments