Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Python 2.6's multiprocessing lock not working on second use?

Reply
Thread Tools

Python 2.6's multiprocessing lock not working on second use?

 
 
Frédéric Sagnes
Guest
Posts: n/a
 
      01-16-2009
Hi,

I ran a few tests on the new Python 2.6 multiprocessing module before
migrating a threading code, and found out the locking code is not
working well. In this case, a pool of 5 processes is running, each
trying to get the lock and releasing it after waiting 0.2 seconds
(action is repeated twice). It looks like the multiprocessing lock
allows multiple locking after the second pass. Running the exact same
code with threads works correctly.

Further down is the test code, output is great when running with
threads (the sequence of lock/unlock looks good), but the output gets
mixed up (mutliple locks in a row) when running with processes.

My setup is : Mac OS X 10.5 running Python 2.6.1 from MacPython

Did I do something wrong, or is there a limitation for multiprocessing
locks that I am not aware of?

Thank you for your help!

-- Fred

-------------------------------

#!/usr/bin/python
# -*- coding: utf-8 -*-

from multiprocessing import Process, Queue, Lock
from Queue import Empty
from threading import Thread
import time

class test_lock_process(object):
def __init__(self, lock, id, queue):
self.lock = lock
self.id = id
self.queue = queue
self.read_lock()

def read_lock(self):
for i in xrange(2):
self.lock.acquire()
self.queue.put('[proc%d] Got lock' % self.id)
time.sleep(.2)
self.queue.put('[proc%d] Released lock' % self.id)
self.lock.release()

def test_lock(processes=10, lock=Lock(), process=True, queue=None):
print_result = False
if queue == None:
print_result = True
queue = Queue()

threads = []
for i in xrange(processes):
if process: threads.append(Process(target=test_lock_process,
args=(lock,i,queue,)))
else: threads.append(Thread(target=test_lock_process, args=
(lock,i,queue,)))

for t in threads:
t.start()

for t in threads:
t.join()

if print_result:
try:
while True: print queue.get(block=False)
except Empty:
pass

if __name__ == "__main__":
#test_lock(processes=5, process=True)
test_lock(processes=5)
 
Reply With Quote
 
 
 
 
Gabriel Genellina
Guest
Posts: n/a
 
      01-17-2009
En Fri, 16 Jan 2009 14:41:21 -0200, escribiste en el grupo
gmane.comp.python.general

> I ran a few tests on the new Python 2.6 multiprocessing module before
> migrating a threading code, and found out the locking code is not
> working well. In this case, a pool of 5 processes is running, each
> trying to get the lock and releasing it after waiting 0.2 seconds
> (action is repeated twice). It looks like the multiprocessing lock
> allows multiple locking after the second pass. Running the exact same
> code with threads works correctly.


I've tested your code on Windows and I think the problem is on the Queue
class. If you replace the Queue with some print statements or write to a
log file, the sequence lock/release is OK.
You should file a bug report on http://bugs.python.org/

--
Gabriel Genellina

 
Reply With Quote
 
 
 
 
Frédéric Sagnes
Guest
Posts: n/a
 
      01-19-2009
On Jan 17, 11:32*am, "Gabriel Genellina" <(E-Mail Removed)>
wrote:
> En Fri, 16 Jan 2009 14:41:21 -0200, escribiste en el grupo
> gmane.comp.python.general
>
> > I ran a few tests on the new Python 2.6multiprocessingmodule before
> > migrating a threading code, and found out the locking code is not
> > working well. In this case, a pool of 5 processes is running, each
> > trying to get the lock and releasing it after waiting 0.2 seconds
> > (action is repeated twice). It looks like themultiprocessinglock
> > allows multiple locking after the second pass. Running the exact same
> > code with threads works correctly.

>
> I've tested your code on Windows and I think the problem is on the Queue
> class. If you replace the Queue with some print statements or write to a
> log file, the sequence lock/release is OK.
> You should file a bug report onhttp://bugs.python.org/
>
> --
> Gabriel Genellina


Thanks for your help gabriel, I just tested it without the queue and
it works! I'll file a bug about the queues.

Fred

For those interested, the code that works (well, it always did, but
this shows the real result):

class test_lock_process(object):
def __init__(self, lock):
self.lock = lock
self.read_lock()

def read_lock(self):
for i in xrange(5):
self.lock.acquire()
logging.info('Got lock')
time.sleep(.2)
logging.info('Released lock')
self.lock.release()

if __name__ == "__main__":
logging.basicConfig(format='[%(process)04d@%(relativeCreated)04d] %
(message)s', level=logging.DEBUG)

lock = Lock()

processes = []
for i in xrange(2):
processes.append(Process(target=test_lock_process, args=
(lock,)))

for t in processes:
t.start()

for t in processes:
t.join()
 
Reply With Quote
 
Frédéric Sagnes
Guest
Posts: n/a
 
      01-19-2009
On Jan 19, 11:53*am, Frédéric Sagnes <(E-Mail Removed)> wrote:
> On Jan 17, 11:32*am, "Gabriel Genellina" <(E-Mail Removed)>
> wrote:
>
>
>
> > En Fri, 16 Jan 2009 14:41:21 -0200, escribiste en el grupo
> > gmane.comp.python.general

>
> > > I ran a few tests on the new Python 2.6multiprocessingmodule before
> > > migrating a threading code, and found out the locking code is not
> > > working well. In this case, a pool of 5 processes is running, each
> > > trying to get the lock and releasing it after waiting 0.2 seconds
> > > (action is repeated twice). It looks like themultiprocessinglock
> > > allows multiple locking after the second pass. Running the exact same
> > > code with threads works correctly.

>
> > I've tested your code on Windows and I think the problem is on the Queue
> > class. If you replace the Queue with some print statements or write to a
> > log file, the sequence lock/release is OK.
> > You should file a bug report onhttp://bugs.python.org/

>
> > --
> > Gabriel Genellina

>
> Thanks for your help gabriel, I just tested it without the queue and
> it works! I'll file a bug about the queues.
>
> Fred
>
> For those interested, the code that works (well, it always did, but
> this shows the real result):
>
> class test_lock_process(object):
> * * def __init__(self, lock):
> * * * * self.lock = lock
> * * * * self.read_lock()
>
> * * def read_lock(self):
> * * * * for i in xrange(5):
> * * * * * * self.lock.acquire()
> * * * * * * logging.info('Got lock')
> * * * * * * time.sleep(.2)
> * * * * * * logging.info('Released lock')
> * * * * * * self.lock.release()
>
> if __name__ == "__main__":
> * * logging.basicConfig(format='[%(process)04d@%(relativeCreated)04d] %
> (message)s', level=logging.DEBUG)
>
> * * lock = Lock()
>
> * * processes = []
> * * for i in xrange(2):
> * * * * processes.append(Process(target=test_lock_process, args=
> (lock,)))
>
> * * for t in processes:
> * * * * t.start()
>
> * * for t in processes:
> * * * * t.join()


Opened issue #4999 [http://bugs.python.org/issue4999] on the matter,
referencing this thread.

--
Frédéric Sagnes
 
Reply With Quote
 
Jesse Noller
Guest
Posts: n/a
 
      01-19-2009
On Mon, Jan 19, 2009 at 8:16 AM, Frédéric Sagnes <(E-Mail Removed)> wrote:
> On Jan 19, 11:53 am, Frédéric Sagnes <(E-Mail Removed)> wrote:
>> On Jan 17, 11:32 am, "Gabriel Genellina" <(E-Mail Removed)>
>> wrote:
>>
>>
>>
>> > En Fri, 16 Jan 2009 14:41:21 -0200, escribiste en el grupo
>> > gmane.comp.python.general

>>
>> > > I ran a few tests on the new Python 2.6multiprocessingmodule before
>> > > migrating a threading code, and found out the locking code is not
>> > > working well. In this case, a pool of 5 processes is running, each
>> > > trying to get the lock and releasing it after waiting 0.2 seconds
>> > > (action is repeated twice). It looks like themultiprocessinglock
>> > > allows multiple locking after the second pass. Running the exact same
>> > > code with threads works correctly.

>>
>> > I've tested your code on Windows and I think the problem is on the Queue
>> > class. If you replace the Queue with some print statements or write to a
>> > log file, the sequence lock/release is OK.
>> > You should file a bug report onhttp://bugs.python.org/

>>
>> > --
>> > Gabriel Genellina

>>
>> Thanks for your help gabriel, I just tested it without the queue and
>> it works! I'll file a bug about the queues.
>>
>> Fred
>>
>> For those interested, the code that works (well, it always did, but
>> this shows the real result):
>>
>> class test_lock_process(object):
>> def __init__(self, lock):
>> self.lock = lock
>> self.read_lock()
>>
>> def read_lock(self):
>> for i in xrange(5):
>> self.lock.acquire()
>> logging.info('Got lock')
>> time.sleep(.2)
>> logging.info('Released lock')
>> self.lock.release()
>>
>> if __name__ == "__main__":
>> logging.basicConfig(format='[%(process)04d@%(relativeCreated)04d] %
>> (message)s', level=logging.DEBUG)
>>
>> lock = Lock()
>>
>> processes = []
>> for i in xrange(2):
>> processes.append(Process(target=test_lock_process, args=
>> (lock,)))
>>
>> for t in processes:
>> t.start()
>>
>> for t in processes:
>> t.join()

>
> Opened issue #4999 [http://bugs.python.org/issue4999] on the matter,
> referencing this thread.
>


Thanks, I've assigned it to myself. Hopefully I can get a fix put
together soonish, time permitting.
-jesse
 
Reply With Quote
 
Jesse Noller
Guest
Posts: n/a
 
      01-19-2009
On Mon, Jan 19, 2009 at 1:32 PM, Nick Craig-Wood <(E-Mail Removed)> wrote:
> Jesse Noller <(E-Mail Removed)> wrote:
>> > Opened issue #4999 [http://bugs.python.org/issue4999] on the matter,
>> > referencing this thread.

>>
>> Thanks, I've assigned it to myself. Hopefully I can get a fix put
>> together soonish, time permitting.

>
> Sounds like it might be hard or impossible to fix to me. I'd love to
> be proved wrong though!
>
> If you were thinking of passing time.time() /
> clock_gettime(CLOCK_MONOTONIC) along in the Queue too, then you'll
> want to know that it can differ by significant amounts on different
> processors
>
> Good luck!
>


Consider my parade rained on. And after looking at it this morning,
yes - this is going to be hard, and should be fixed for a FIFO queue
:\

-jesse
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
When to lock data items returned by multiprocessing.Manager? Veloz Python 1 03-15-2010 07:57 PM
Read-Write Lock vs primitive Lock() k3xji Python 7 12-30-2008 10:19 PM
Application.Lock()/UnLock() or lock(Application) nano2k ASP .Net Web Services 2 08-09-2007 09:31 AM
Threading - Why Not Lock Objects Rather than lock the interpreter Fuzzyman Python 3 12-05-2003 10:43 PM
RE: Threading - Why Not Lock Objects Rather than lock theinterpreter Robert Brewer Python 0 12-05-2003 05:33 PM



Advertisments