«

Hello,

Is anyone has experiance in running python code to run multi thread
parallel in multi processor. Is it possible ?

Can python manage which cpu shoud do every thread?

Sincerely Yours,
Pujo

a écrit :
> Hello,
>
> Is anyone has experiance in running python code to run multi thread
> parallel in multi processor. Is it possible ?
>
> Can python manage which cpu shoud do every thread?
>
> Sincerely Yours,
> Pujo
>

There's just no way you can use Python in a multi-processor environment,
because the GIL (Global Interpreter Lock) will prevent two threads from
running concurrently. When I saw this discussed, the Python developper
were more into multi-process systems when it comes to multi-processors.
I think I even heard some discussion about efficient inter-process
messaging system, but I can't remember where )

Hope it'll help.

Pierre

Hello Pierre,

That's a pity, since when we have to run parallel, with single
processor is really not efficient. To use more computers I think is
cheaper than to buy super computer in developt country.

Sincerely Yours,
pujo aji

[]
> That's a pity, since when we have to run parallel, with single
> processor is really not efficient. To use more computers I think is
> cheaper than to buy super computer in developt country.

Although cpython has a GIL that prevents multiple python threads *in the
same python process* from running *inside the python interpreter* at the
same time (I/O is not affected, for example), this can be gotten around
by using multiple processes, each bound to a different CPU, and using
some form of IPC (pyro, CORBA, bespoke, etc) to communicate between
those processes.

This solution is not ideal, because it will probably involve
restructuring your app. Also, all of the de/serialization involved in
the IPC will slow things down, unless you're using POSH, a shared memory
based system that requires System V IPC.

http://poshmodule.sf.net

Alternatively, you could simply use either jython or ironpython, both of
which have no central interpreter lock (because they rely on JVM/CLR
garbage collection), and thus will support transparent migration of
threads to multiple processors in a multi-cpu system, if the underlying
VM supports that.

http://www.jython.org
http://www.ironpython.com

And you shouldn't have to restructure your code, assuming that it is
already thread-safe?

For interest, I thought I'd mention PyLinda, a distributed object system
that takes a completely different, higher level, approach to object
distribution: it creates "tuple space", where objects live. The objects
can be located and sent messages. But (Py)Linda hides most of gory
details of how objects actually get distributed, and the mechanics of
actually connecting with those remote objects.

http://www-users.cs.york.ac.uk/~aw/pylinda/

HTH,

--
alan kennedy
------------------------------------------------------
email alan: http://xhaus.com/contact/alan

On Wed, Feb 09, 2005 at 07:56:27AM -0800, wrote:
> Hello Pierre,
>
> That's a pity, since when we have to run parallel, with single
> processor is really not efficient. To use more computers I think is
> cheaper than to buy super computer in developt country.

and buying more, cheap computers gives you more processing power than
buying less, multi-processor computers. So the best thing you can do
is learn to leverage some distributed computing scheme. Take a look at
Pyro, and its Event server.

--
John Lenton () -- Random fortune:
When the cup is full, carry it level.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFCCjiegPqu395ykGsRAhYiAKCa837YZ2F6HK3DeAGOif be9DPs5gCfd5ab
Cadcx1hVe9Jz+GE8DipUdt4=
=xCFM
-----END PGP SIGNATURE-----

In article < .com>,
<> wrote:
>
>Is anyone has experiance in running python code to run multi thread
>parallel in multi processor. Is it possible ?

Yes. The GIL prevents multiple Python threads from running
simultaneously, but C extensions can release the GIL; all I/O functions
in the standard library do, so threading Python makes sense for e.g. web
spiders. See the slides on my website for more info.

>Can python manage which cpu shoud do every thread?

Nope.
--
Aahz () <*> http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code --
not in reams of trivial code that bores the reader to death." --GvR

John Lenton <> writes:
> and buying more, cheap computers gives you more processing power than
> buying less, multi-processor computers.

The day is coming when even cheap computers have multiple cpu's.
See hyperthreading and the coming multi-core P4's, and the finally
announced Cell processor.

Conclusion: the GIL must die.

Paul Rubin wrote:
> John Lenton <> writes:
>
>>and buying more, cheap computers gives you more processing power than
>>buying less, multi-processor computers.
>
>
> The day is coming when even cheap computers have multiple cpu's.
> See hyperthreading and the coming multi-core P4's, and the finally
> announced Cell processor.
>
> Conclusion: the GIL must die.

I think I agree with this.
And I couldn't resist:
http://www.razorvine.net/img/GIL.jpg



--Irmen

In article <>,
Paul Rubin <http://> wrote:
>
>The day is coming when even cheap computers have multiple cpu's.
>See hyperthreading and the coming multi-core P4's, and the finally
>announced Cell processor.
>
>Conclusion: the GIL must die.

It's not clear to what extent these processors will perform well with
shared memory space. One of the things I remember most about Bruce
Eckel's discussions of Java and threading is just how broken Java's
threading model is in certain respects when it comes to CPU caches
failing to maintain cache coherency. It's always going to be true that
getting fully scaled performance will require more CPUs with non-shared
memory -- that's going to mean IPC with multiple processes instead of
threads.

Don't get me wrong -- I'm probably one of the bigger boosters of threads.
But it bugs me when people think that getting rid of the GIL will be the
Holy Grail of Python performance. No way. No how. No time.
--
Aahz () <*> http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code --
not in reams of trivial code that bores the reader to death." --GvR

On Sat, Feb 12, 2005 at 07:13:17PM -0500, Aahz wrote:
> In article <>,
> Paul Rubin <http://> wrote:
> >
> >The day is coming when even cheap computers have multiple cpu's.
> >See hyperthreading and the coming multi-core P4's, and the finally
> >announced Cell processor.

I'm looking forward to Multi-core P4s (and Opterons). The Cell is a
non-starter for general purpose computing. Arstechnica has a couple
good pieces on it, the upshot is that it is one normal processor
with eight strange floating point co-processors hanging off it.

> >Conclusion: the GIL must die.
>
> It's not clear to what extent these processors will perform well with
> shared memory space. One of the things I remember most about Bruce
> Eckel's discussions of Java and threading is just how broken Java's
> threading model is in certain respects when it comes to CPU caches
> failing to maintain cache coherency. It's always going to be true that
> getting fully scaled performance will require more CPUs with non-shared
> memory -- that's going to mean IPC with multiple processes instead of
> threads.
>
> Don't get me wrong -- I'm probably one of the bigger boosters of threads.
> But it bugs me when people think that getting rid of the GIL will be the
> Holy Grail of Python performance. No way. No how. No time.

"Me Too!" for a small number of processors (four) it is easier (and
usually sufficient) to pipeline functional parts into different
processes than it is to thread the whole monkey. As a bonus this usually
gives you scaling across machines (and not just CPUs) for cheap. I'm
aware there are some problems where this isn't true. From reading this
thread every couple months on c.l.py for the last few years it is my
opinion that the number of people who think threading is the only solution
to their problem greatly outnumber the number of people who actually have
such a problem (like, nearly all of them).

Killing the GIL is proposing a silver bullet where there is no werewolf-ly,

-Jack