Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > multiprocessing

Reply
Thread Tools

multiprocessing

 
 
elsa
Guest
Posts: n/a
 
      04-07-2011
Hi guys,

I want to try out some pooling of processors, but I'm not sure if it
is possible to do what I want to do. Basically, I want to have a
global object, that is updated during the execution of a function, and
I want to be able to run this function several times on parallel
processors. The order in which the function runs doesn't matter, and
the value of the object doesn't matter to the function, but I do want
the processors to take turns 'nicely' when updating the object, so
there are no collisions. Here is an extremely simplified and trivial
example of what I have in mind:

from multiprocessing import Pool
import random

p=Pool(4)
myDict={}

def update(value):
global myDict
index=random.random()
myDict[index]+=value

total=1000

p.map(update,range(total))


After, I would also like to be able to use several processors to
access the global object (but not modify it). Again, order doesn't
matter:

p1=Pool(4)

def getValues(index):
global myDict
print myDict[index]

p1.map(getValues,keys.myDict)

Is there a way to do this?

Thanks,

Elsa.

 
Reply With Quote
 
 
 
 
Philip Semanchuk
Guest
Posts: n/a
 
      04-08-2011

On Apr 7, 2011, at 8:57 PM, Kerensa McElroy wrote:

>
> Hi,
>
> thanks for your response.
>
> I checked out multiprocessing.value, however from what I can make out, it works with object of only a very limited type. Is there a way to do this for more complex objects? (In reality, my object is a large multi-dimensional numpy array).


Elsa,
Are you following the current thread in this list which is talking about sharing numpy arrays via multiprocessing?

http://mail.python.org/pipermail/pyt...l/1269173.html





> Date: Wed, 6 Apr 2011 22:20:06 -0700
> Subject: Re: multiprocessing
> From: http://www.velocityreviews.com/forums/(E-Mail Removed)
> To: (E-Mail Removed)
> CC: (E-Mail Removed)
>
>
> On Wed, Apr 6, 2011 at 9:06 PM, elsa <(E-Mail Removed)> wrote:
>
> Hi guys,
>
>
>
> I want to try out some pooling of processors, but I'm not sure if it
>
> is possible to do what I want to do. Basically, I want to have a
>
> global object, that is updated during the execution of a function, and
>
> I want to be able to run this function several times on parallel
>
> processors. The order in which the function runs doesn't matter, and
>
> the value of the object doesn't matter to the function, but I do want
>
> the processors to take turns 'nicely' when updating the object, so
>
> there are no collisions. Here is an extremely simplified and trivial
>
> example of what I have in mind:
>
>
>
> from multiprocessing import Pool
>
> import random
>
>
>
> p=Pool(4)
>
> myDict={}
>
>
>
> def update(value):
>
> global myDict
>
> index=random.random()
>
> myDict[index]+=value
>
>
>
> total=1000
>
>
>
> p.map(update,range(total))
>
>
>
>
>
> After, I would also like to be able to use several processors to
>
> access the global object (but not modify it). Again, order doesn't
>
> matter:
>
>
>
> p1=Pool(4)
>
>
>
> def getValues(index):
>
> global myDict
>
> print myDict[index]
>
>
>
> p1.map(getValues,keys.myDict)
>
>
>
> Is there a way to do this
> This should give you a synchronized wrapper around an object in shared memory:
>
> http://docs.python.org/library/multi...ocessing.Value
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Using multiprocessing nhwarriors Python 4 10-11-2008 04:48 AM
2.6 multiprocessing and pdb Aaron \Castironpi\ Brady Python 1 10-03-2008 08:05 AM
multiprocessing eats memory Max Ivanov Python 6 09-27-2008 10:29 PM
multiprocessing module (PEP 371) sturlamolden Python 6 06-08-2008 04:33 AM
Parallel/Multiprocessing script design question Amit N Python 4 09-13-2007 08:07 PM



Advertisments