Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Embedding Python, threading and scalability

Reply
Thread Tools

Re: Embedding Python, threading and scalability

 
 
Jeff Epler
Guest
Posts: n/a
 
      07-10-2003
On Thu, Jul 10, 2003 at 11:14:47AM +0800, Simon Wittber (Maptek) wrote:
> Seriously though, this is an issue, which is a major hurdle Python *has*
> to cross if it is ever going to be seriously considered for use on large
> projects, on large SMP hardware.


Has anybody proposed "how to get there from here" for this problem
(useful multithreading of non-blocking pure Python code)? I'm not
bright enough to see how, that's for sure. Especially if you are
talking about an incremental approach, not the mythical "Python 3000".
What I mean is that you have to work this magic *and* somehow let
existing modules written in C work, with at worst a recompile. (as
someone who doesn't *need* threads, but works on a project with piles of
Python modules written in C, that's my bias anyway)

Someone nearby mentioned lua, but I don't know what it did for threading.
Perl and tcl both seem to have taken the approach of having each thread
be a distinct interpreter with nothing shared. While this means you
never have to worry about locking against a reader or modifier in another
thread, it means you might as well be using the processes that the angry
Unix Gods gave us in the first place. <1/3 overstatement> I'm pretty sure
that this approach has been explicitly ruled out by that other angry God,
Guido van Rossum, anyway.

I've written Python programs that use threads in a way that was expedient
to get a user interface running without long seizures, while I've never
written a thread in tcl or perl. OTOH if I'd had to treat everything
as explicit message passing (a la tcl threading), I'd have just found
another way (like "after idle" and "update" in tcl)

Jython will give you Java's thread model today, for Python code, won't
it? Back when I benchmarked it, Jython code and Python code ran at
fairly similar speeds (in pybench), if only you had a machine that could
keep the Jython interpreter from swapping...

Jeff

 
Reply With Quote
 
 
 
 
Donn Cave
Guest
Posts: n/a
 
      07-10-2003
Quoth Jeff Epler <(E-Mail Removed)>:
| On Thu, Jul 10, 2003 at 11:14:47AM +0800, Simon Wittber (Maptek) wrote:
|> Seriously though, this is an issue, which is a major hurdle Python *has*
|> to cross if it is ever going to be seriously considered for use on large
|> projects, on large SMP hardware.
|
| Has anybody proposed "how to get there from here" for this problem
| (useful multithreading of non-blocking pure Python code)? I'm not
| bright enough to see how, that's for sure. Especially if you are
| talking about an incremental approach, not the mythical "Python 3000".
| What I mean is that you have to work this magic *and* somehow let
| existing modules written in C work, with at worst a recompile. (as
| someone who doesn't *need* threads, but works on a project with piles of
| Python modules written in C, that's my bias anyway)

Ha - I knew I'd find an answer for this if I searched for "free threading" -
and among other hits, I found this very thread! So in Andrew Dalke's words:

> Ahh, the Global Interpreter Lock (GIL).
>
> Years ago, Greg Stein had a version of Python 1.4 running with no GIL.
>
> http://www.python.org/ftp/python/con...reading.README
>
> Search for "free threading" to get more hits on this topic.


At any rate, it sure isn't because Guido can't scare up an SMP machine.

| I've written Python programs that use threads in a way that was expedient
| to get a user interface running without long seizures, while I've never
| written a thread in tcl or perl. OTOH if I'd had to treat everything
| as explicit message passing (a la tcl threading), I'd have just found
| another way (like "after idle" and "update" in tcl)

Hm, not sure what you're alluding to here. I actually kind of like an
I/O thread dispatch model myself, but I don't think Python cares either
way nor would it if the GIL were abolished.

| Jython will give you Java's thread model today, for Python code, won't
| it? Back when I benchmarked it, Jython code and Python code ran at
| fairly similar speeds (in pybench), if only you had a machine that could
| keep the Jython interpreter from swapping...

Don't know, but I noticed in another thread among the hits that Ype Kingma
asserts that Jython does indeed support fine grained multithreading.

Donn Cave, http://www.velocityreviews.com/forums/(E-Mail Removed)
 
Reply With Quote
 
 
 
 
Peter Hansen
Guest
Posts: n/a
 
      07-10-2003
Afanasiy wrote:
>
> On Thu, 10 Jul 2003 05:26:14 -0000, "Donn Cave" <(E-Mail Removed)> wrote:
>
> >Quoth Jeff Epler <(E-Mail Removed)>:

>
> >At any rate, it sure isn't because Guido can't scare up an SMP machine.

>
> Is this the absolute truth or just something people like to say?
>
> If it is true I am very surprised... I think getting Guido an SMP
> machine will not be very difficult, and could well be very cheap.


Note the double negative in the phrase "isn't because Guido can't".

That means in effect "Guido can". Nobody's disagreeing with that.

-Peter
 
Reply With Quote
 
Afanasiy
Guest
Posts: n/a
 
      07-10-2003
On Thu, 10 Jul 2003 15:36:04 -0400, Peter Hansen <(E-Mail Removed)>
wrote:

>Afanasiy wrote:
>>
>> On Thu, 10 Jul 2003 05:26:14 -0000, "Donn Cave" <(E-Mail Removed)> wrote:
>>
>> >Quoth Jeff Epler <(E-Mail Removed)>:

>>
>> >At any rate, it sure isn't because Guido can't scare up an SMP machine.

>>
>> Is this the absolute truth or just something people like to say?
>>
>> If it is true I am very surprised... I think getting Guido an SMP
>> machine will not be very difficult, and could well be very cheap.

>
>Note the double negative in the phrase "isn't because Guido can't".
>
>That means in effect "Guido can". Nobody's disagreeing with that.


Yes, I read a previous post which said "Give Guido a SMP machine."
and read this one as the same, my mistake.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Performance and Scalability of JSP and ASP tharma ASP .Net 4 09-13-2005 06:04 PM
Embedding Python in Multi-threading App. Any Guidelines, Hints, Recipes ?? adsheehan@eircom.net Python 0 06-07-2005 03:12 PM
Question: Threading and embedding python in an application David Harrison Python 3 05-10-2005 08:44 PM
Embedding Python, threading and scalability Wenning Qiu Python 7 07-11-2003 08:22 PM
RE: Embedding Python, threading and scalability Simon Wittber (Maptek) Python 7 07-11-2003 06:07 PM



Advertisments