Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: [Python-Dev] Making python C-API thread safe (try 2)

Reply
Thread Tools

Re: [Python-Dev] Making python C-API thread safe (try 2)

 
 
Christopher A. Craig
Guest
Posts: n/a
 
      09-12-2003
Harri Pesonen <(E-Mail Removed)> writes:

> After sleeping over night, I think that I got it. The simple solution is,
> that each thread created in Python gets its own independent interpreter state
> as well. And there could be a separate thread-global interpreter state for
> shared memory access. Access to this global state would always be
> synchronized. There could even be multiple named global states, so that the
> thread interlocking could be minimized. The python syntax for creating objects
> in this global state should be invented:
>
>
> synchronize a = "abcd"
>
> Also when creating the new thread, it's arguments would be copied from the
> creating state to the new state.
>
>
> What does it sound? Of course it would be incompatible with the current
> threading system in Python, but it would be totally multithreading, no global
> interpreter lock needed. It would be faster than current Python, there would
> be no need to free or acquire the lock when calling OS functions, and no need
> to check how many byte codes have been processed, etc.
>


Couldn't you do this now with multiple processes and the shm module?

--
Christopher A. Craig <(E-Mail Removed)>
"I affirm brethren by the boasting in you which I have in Christ Jesus
our Lord, I die daily" I Cor 15:31 (NASB)


 
Reply With Quote
 
 
 
 
Alan Kennedy
Guest
Posts: n/a
 
      09-12-2003
Harri Pesonen <(E-Mail Removed)> writes:
>> The simple solution is,
>> that each thread created in Python gets its own independent
>> interpreter state as well. And there could be a separate
>> thread-global interpreter state for shared memory access. Access
>> to this global state would always be
>> synchronized.


"Christopher A. Craig" wrote:
> Couldn't you do this now with multiple processes and the shm module?


Yes, you can store representations of python objects in shared memory,
e.g. pickles, etc: an in-memory database, with indices, etc.

So when you want to access a "shared" object, you go to the shared
store, retrieve the pickle, unpickle it and use it, making sure to
repickle any changes and store them back in the shared memory. Which
might be cumbersome and inefficient, particularly if your goal is to
maximise efficiency in a multi-processor situation.

But if you want to store actual python objects in shared memory, and
avoid all the pickling etc,then you have to change the interpreter so
that it obtains the memory for new python objects from the shared
memory pool instead of "local" memory.

Which leads to problems with reference counting and garbage
collection. These would have to take multiple processes into account:
what happens when a process goes down? Should the objects allocated by
it be destroyed? Or remain persistent? Until when? If a process died
in an unclean fashion, it might not delete its references to objects
cleanly.

And it gets more complex again when you're dealing with users and
permissions.

I think that there are so many questions associated with the approach
of sharing python objects through shared memory that it will probably
remain an "application specific" technique for some time to come.

Though doubtless some more knowledgable person than I will now
contradict me. Please.

--
alan kennedy
-----------------------------------------------------
check http headers here: http://xhaus.com/headers
email alan: http://xhaus.com/mailto/alan
 
Reply With Quote
 
 
 
 
Shane Hathaway
Guest
Posts: n/a
 
      09-12-2003
Alan Kennedy wrote:
> Which leads to problems with reference counting and garbage
> collection. These would have to take multiple processes into account:
> what happens when a process goes down? Should the objects allocated by
> it be destroyed? Or remain persistent? Until when? If a process died
> in an unclean fashion, it might not delete its references to objects
> cleanly.
>
> And it gets more complex again when you're dealing with users and
> permissions.
>
> I think that there are so many questions associated with the approach
> of sharing python objects through shared memory that it will probably
> remain an "application specific" technique for some time to come.
>
> Though doubtless some more knowledgable person than I will now
> contradict me. Please.


I'd just like to point out that Steffen Viken Valvag has just recently
(March 2003) researched this, found solutions, and written an
implementation.

http://poshmodule.sourceforge.net/posh/html/node4.html

That page leaves a few questions about the distributed garbage
collection unanswered, but it's a nice high-level overview.

I don't know how stable Posh is, but if nothing else it is a convincing
proof of concept.

Shane


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
html5lib not thread safe. Is the Python SAX library thread-safe? John Nagle Python 5 03-12-2012 04:07 PM
os.ChDir() not thread-safe; was : Is tempfile.mkdtemp() thread-safe? Gabriel Rossetti Python 0 08-29-2008 08:30 AM
RE: [Python-Dev] Making python C-API thread safe (try 2) Brian Quinlan Python 12 09-17-2003 02:23 PM
Re: [Python-Dev] Making python C-API thread safe (try 2) Harri Pesonen Python 32 09-17-2003 02:08 PM
Re: [Python-Dev] Making python C-API thread safe (try 2) Shane Hathaway Python 3 09-12-2003 05:50 AM



Advertisments