Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Multiprocessing, shared memory vs. pickled copies

Reply
Thread Tools

Re: Multiprocessing, shared memory vs. pickled copies

 
 
Miki Tebeka
Guest
Posts: n/a
 
      04-10-2011
> Now, I don't know that I actually HAVE to pass my neural network and
> input data as copies -- they're both READ-ONLY objects for the
> duration of an evaluate function (which can go on for quite a while).

One option in that case is to use "fork" (if you're on a *nix machine).
See http://pythonwise.blogspot.com/2009/04/pmap.html for example

HTH
--
Miki Tebeka <(E-Mail Removed)>
http://pythonwise.blogspot.com
 
Reply With Quote
 
 
 
 
John Nagle
Guest
Posts: n/a
 
      04-10-2011
On 4/10/2011 9:11 AM, Miki Tebeka wrote:
>> Now, I don't know that I actually HAVE to pass my neural network and
>> input data as copies -- they're both READ-ONLY objects for the
>> duration of an evaluate function (which can go on for quite a while).

> One option in that case is to use "fork" (if you're on a *nix machine).
> See http://pythonwise.blogspot.com/2009/04/pmap.html for example


Unless you have a performance problem, don't bother with shared
memory.

If you have a performance problem, Python is probably the wrong
tool for the job anyway.

John Nagle
 
Reply With Quote
 
 
 
 
sturlamolden
Guest
Posts: n/a
 
      04-10-2011
On 10 apr, 18:27, John Nagle <(E-Mail Removed)> wrote:

> * * Unless you have a performance problem, don't bother with shared
> memory.
>
> * * If you have a performance problem, Python is probably the wrong
> tool for the job anyway.


Then why does Python have a multiprocessing module?

In my opinion, if Python has a multiprocessing module in the standard
library, it should also be possible to use it with NumPy.

Sturla
 
Reply With Quote
 
John Nagle
Guest
Posts: n/a
 
      04-11-2011
On 4/10/2011 3:29 PM, sturlamolden wrote:
> On 10 apr, 18:27, John Nagle<(E-Mail Removed)> wrote:
>
>> Unless you have a performance problem, don't bother with shared
>> memory.
>>
>> If you have a performance problem, Python is probably the wrong
>> tool for the job anyway.

>
> Then why does Python have a multiprocessing module?


Because nobody can fix the Global Interpreter Lock problem in CPython.

The multiprocessing module is a hack to get around the fact
that Python threads don't run concurrently, and thus, threaded
programs don't effectively use multi-core CPUs.

John Nagle

 
Reply With Quote
 
sturlamolden
Guest
Posts: n/a
 
      04-11-2011
On 11 apr, 09:21, John Nagle <(E-Mail Removed)> wrote:

> * * *Because nobody can fix the Global Interpreter Lock problem in CPython.
>
> * * *The multiprocessing module is a hack to get around the fact
> that Python threads don't run concurrently, and thus, threaded
> programs don't effectively use multi-core CPUs.


Then why not let NumPy use shared memory in this case? It is as simple
as this:

import numpy as np
import sharedmem as sm
private_array = np.zeros((10,10))
shared_array = sm.zeros((10,10))

I.e. the code is identical (more or less).

Also, do you think shared memory has other uses besides parallel
computing, such as IPC? It is e.g. an easy way to share data between
Python and an external program. Do you e.g. need to share (not send)
large amounts of data between Python and Java or Matlab?

The difference here from multiprocessing.Array is that IPC with
multiprocessing.Queue will actually work, because I am not using
anonymous shared memory. Instead of inherting handles, the segment has
a name in the file system, which means it can be memory mapped from
anywhere.

I posted the code to the NumPy mailing list yesterday.

Sturla
 
Reply With Quote
 
sturlamolden
Guest
Posts: n/a
 
      04-11-2011
On 11 apr, 21:11, sturlamolden <(E-Mail Removed)> wrote:

> import numpy as np
> import sharedmem as sm
> private_array = np.zeros((10,10))
> shared_array *= sm.zeros((10,10))


I am also using this to implement synchronization primitives (barrier,
lock, semaphore, event) that can be sent over an instance of
multiprocessing.Queue. The synchronization primitives in
multiprocessing cannot be communicated this way.

Sturla
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Multiprocessing, shared memory vs. pickled copies John Ladasky Python 21 04-10-2011 10:35 PM
within a class, redefining self with pickled file syd Python 2 04-08-2005 05:47 AM
Pickled text file causing ValueError (dos/unix issue) Aki Niimura Python 12 01-15-2005 04:05 PM
how do I read a series of pickled objects from a socket? Ryan Grow Python 1 11-16-2004 09:17 PM
recover a partly destroyed pickled structure Maurizio Berlusconi Python 1 09-25-2004 05:28 PM



Advertisments