Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > numarray and SMP

Reply
Thread Tools

numarray and SMP

 
 
Christopher T King
Guest
Posts: n/a
 
      07-01-2004
In a quest to speed up numarray computations, I tried writing a 'threaded
array' class for use on SMP systems that would distribute its workload
across the processors. I hit a snag when I found out that since the Python
interpreter is not reentrant, this effectively disables parallel
processing in Python. I've come up with two solutions to this problem,
both involving numarray's C functions that perform the actual vector
operations:

1) Surround the C vector operations with Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS, thus allowing the vector operations (which don't
access Python structures) to run in parallel with the interpreter.
Python glue code would take care of threading and locking.

2) Move the parallelization into the C vector functions themselves. This
would likely get poorer performance (a chain of vector operations
couldn't be combined into one threaded operation).

I'd much rather do #1, but will playing around with the interpreter state
like that cause any problems?

 
Reply With Quote
 
 
 
 
Fernando Perez
Guest
Posts: n/a
 
      07-01-2004
Christopher T King wrote:

> In a quest to speed up numarray computations, I tried writing a 'threaded
> array' class for use on SMP systems that would distribute its workload
> across the processors. I hit a snag when I found out that since the Python
> interpreter is not reentrant, this effectively disables parallel
> processing in Python. I've come up with two solutions to this problem,
> both involving numarray's C functions that perform the actual vector
> operations:


[...]

I suggest you repost this to the numpy list as well. Not only are the
developers there, but this issue interests many of us, so you'd get an eager
audience and more discussion. Not that I don't think c.l.py is a good forum,
quite the contrary: many threading experts live here and not in numpy. I
meant posting to both places, since the combined expertise of 'generic
threading' experts from c.l.py and numeric/numarray users/developers will
likely be a good thing.

Best,

f
 
Reply With Quote
 
 
 
 
Christopher T King
Guest
Posts: n/a
 
      07-01-2004
On Thu, 1 Jul 2004, Fernando Perez wrote:

> I suggest you repost this to the numpy list as well. Not only are the
> developers there, but this issue interests many of us, so you'd get an eager
> audience and more discussion. Not that I don't think c.l.py is a good forum,
> quite the contrary: many threading experts live here and not in numpy. I
> meant posting to both places, since the combined expertise of 'generic
> threading' experts from c.l.py and numeric/numarray users/developers will
> likely be a good thing.


Thanks, and done.

 
Reply With Quote
 
Fernando Perez
Guest
Posts: n/a
 
      07-01-2004
Christopher T King wrote:

> On Thu, 1 Jul 2004, Fernando Perez wrote:
>
>> I suggest you repost this to the numpy list as well. Not only are the

[...]
>
> Thanks, and done.


Just saw it. I don't really have an answer for you, but I'm curious about what
others on that list may say. We'll see.

Cheers,

f
 
Reply With Quote
 
Aahz
Guest
Posts: n/a
 
      07-06-2004
In article <(E-Mail Removed)>,
Christopher T King <(E-Mail Removed)> wrote:
>
>1) Surround the C vector operations with Py_BEGIN_ALLOW_THREADS and
> Py_END_ALLOW_THREADS, thus allowing the vector operations (which don't
> access Python structures) to run in parallel with the interpreter.
> Python glue code would take care of threading and locking.
>
>2) Move the parallelization into the C vector functions themselves. This
> would likely get poorer performance (a chain of vector operations
> couldn't be combined into one threaded operation).
>
>I'd much rather do #1, but will playing around with the interpreter state
>like that cause any problems?


Not at all -- that's precisely what they're there for. All you have to
do is make sure you're not calling back into Python before doing
Py_END_ALLOW_THREADS. Traditionally, Python has only done this for I/O
operations; I'm very happy to see someone trying to take a whack at the
computational side. (Although I ended up mostly dropping the ball, that
was the original impetus for the Decimal project, to encourage more
computational threading.)
--
Aahz ((E-Mail Removed)) <*> http://www.pythoncraft.com/

"Typing is cheap. Thinking is expensive." --Roy Smith, c.l.py
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
SMP, GIL and Threads catsup Python 11 12-17-2005 05:14 PM
SMP MPP VLIW for machine code Java, Forth, C, Scheme, etc. Reply7471859353@wmconnect.com Java 1 07-31-2005 05:52 AM
SMP MPP VLIW for machine code Java, Forth, C, Scheme, etc. Reply7471859353@wmconnect.com Java 1 07-23-2005 02:33 AM
ASP.net, Windows 2000, and SMP/Multiprocessor Ohaya ASP .Net 0 02-22-2004 03:33 AM
Difficult Python thread problem on AIX SMP machine Gardner Pomper Python 0 11-12-2003 03:09 PM



Advertisments