Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > using python interpreters per thread in C++ program

Reply
Thread Tools

using python interpreters per thread in C++ program

 
 
Ulrich Eckhardt
Guest
Posts: n/a
 
      09-07-2009
ganesh wrote:
>> Did you remeber to acquire the GIL? The GIL is global to the process

>
> No, I did not use GIL.
>
> -- Why do we need to use GIL even though python is private to each
> thread?


Quoting from above: "The GIL is global to the process". So no, it is NOT
private to each thread which means "python" isn't either.

At least that is my understanding of the issue.

Uli

--
Sator Laser GmbH
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932

 
Reply With Quote
 
 
 
 
ganesh
Guest
Posts: n/a
 
      09-07-2009
Actually, I modified my program to have a single shared Py-interpreter
across all threads to test the usage of GIL. So, I did Py_Initialize
in main() function and only called that python function in different
threads.

But this is not the way I want to use interpreters in my code.

I am looking for using sub-interpreters, only that I did not know this
this term, till you mentioned it here.
So bvefore this, I was calling py_Initialize in each of the C++ level
threads, which was wrong.

I should be using Py_NewInterpreter() in my threads and Py_Initialize
() in main() thread. Please correct me if I am wrong.

>>>>not something you want to do unless you want to understand Python C API internals very well.

Are these APIs really tricky to use with C++ embedding? Can't they be
implemented thru C APIs?

I need to use these to get the proper concurrency in my multi-threaded
application without any synchronization mechanisms.
 
Reply With Quote
 
 
 
 
sturlamolden
Guest
Posts: n/a
 
      09-07-2009
On 7 Sep, 13:53, ganesh <(E-Mail Removed)> wrote:

> I need to use these to get the proper concurrency in my multi-threaded
> application without any synchronization mechanisms.


Why will multiple interpreters give you better concurrency? You can
have more than one thread in the same interpreter.

Here is the API explained:

http://docs.python.org/c-api/init.html
http://www.linuxjournal.com/article/3641
 
Reply With Quote
 
sturlamolden
Guest
Posts: n/a
 
      09-07-2009
On 7 Sep, 13:17, Ulrich Eckhardt <(E-Mail Removed)> wrote:

> Quoting from above: "The GIL is global to the process". So no, it is NOT
> private to each thread which means "python" isn't either.
>
> At least that is my understanding of the issue.


Strictly speaking, the GIL is global to the Python DLL, not the
process. You can load it multiple times from different paths, and if
you do GIL with not be shared.

If you make several clean installs of Python on Windows (say c:
\Python26-0 ... c:\Python26-3), you can embed multiple interpreters in
a process, and they will not share GIL or anything else. But be
careful: all *.pyd files must reside in these directories, or they
will be loaded once and refcounts screw up. The Python DLL must also
be in these directories, not in c:\windows\system32. It is in fact
possible to make Python utilize dual- and quad-core processors on
Windows this way. You can even use ctypes for embedding Python into
Python, so no C is required. See:

http://groups.google.no/group/comp.l...2013f9ef3a766d

To make this hack really work one need multiple complete installs, not
just copies of one DLL as I assumed in the thread. But the general
method is the same.

You can see this as a form of primitive object orientation


Sturla Molden

















 
Reply With Quote
 
MRAB
Guest
Posts: n/a
 
      09-07-2009
sturlamolden wrote:
> On 7 Sep, 13:53, ganesh <(E-Mail Removed)> wrote:
>
>> I need to use these to get the proper concurrency in my multi-threaded
>> application without any synchronization mechanisms.

>
> Why will multiple interpreters give you better concurrency? You can
> have more than one thread in the same interpreter.
>
> Here is the API explained:
>
> http://docs.python.org/c-api/init.html
> http://www.linuxjournal.com/article/3641


CPython's GIL means that multithreading on multiple processors/cores has
limitations. Each interpreter has its own GIL, so processor-intensive
applications work better using the multiprocessing module than with the
threading module.
 
Reply With Quote
 
sturlamolden
Guest
Posts: n/a
 
      09-07-2009
On 7 Sep, 14:50, MRAB <(E-Mail Removed)> wrote:

> CPython's GIL means that multithreading on multiple processors/cores has
> limitations. Each interpreter has its own GIL, so processor-intensive
> applications work better using the multiprocessing module than with the
> threading module.


We incur a 200x speed-penalty from Python if the code is CPU-bound.
How much do you gain from one extra processor? Start by adding in some
C, and you will gain much more performance, even without parallel
processing.

Processor-intensive code should therefore use extension libraries that
release the GIL. Then you have option of getting parallel concurrency
by Python threads or OpenMP in C/Fortran. Most processor-intensive
code depend on numerical libraries written in C or Fortran anyway
(NumPy, ATLAS, LAPACK, FFTW, GSL, MKL, etc.) When you do this, the GIL
does not get in your way. The GIL is infact an advantage if one
library happen not to be thread safe. Many old Fortran 77 libraries
have functions with re-entrancy issues, due to SAVE attribute on
variables. Thus, the GIL is often an advantage for processor-intensive
code.

multiprocessing is fragile and unreliable, btw.










 
Reply With Quote
 
Mark Hammond
Guest
Posts: n/a
 
      09-07-2009
On 7/09/2009 10:50 PM, MRAB wrote:
> sturlamolden wrote:
>> On 7 Sep, 13:53, ganesh <(E-Mail Removed)> wrote:
>>
>>> I need to use these to get the proper concurrency in my multi-threaded
>>> application without any synchronization mechanisms.

>>
>> Why will multiple interpreters give you better concurrency? You can
>> have more than one thread in the same interpreter.
>>
>> Here is the API explained:
>>
>> http://docs.python.org/c-api/init.html
>> http://www.linuxjournal.com/article/3641

>
> CPython's GIL means that multithreading on multiple processors/cores has
> limitations. Each interpreter has its own GIL, so processor-intensive
> applications work better using the multiprocessing module than with the
> threading module.


I believe you will find the above is incorrect - even with multiple
interpreter states you still have a single GIL.

Mark

 
Reply With Quote
 
Benjamin Kaplan
Guest
Posts: n/a
 
      09-07-2009
On Mon, Sep 7, 2009 at 6:31 PM, Mark Hammond<(E-Mail Removed)> wrote:
> On 7/09/2009 10:50 PM, MRAB wrote:
>>
>> sturlamolden wrote:
>>>
>>> On 7 Sep, 13:53, ganesh <(E-Mail Removed)> wrote:
>>>
>>>> I need to use these to get the proper concurrency in my multi-threaded
>>>> application without any synchronization mechanisms.
>>>
>>> Why will multiple interpreters give you better concurrency? You can
>>> have more than one thread in the same interpreter.
>>>
>>> Here is the API explained:
>>>
>>> http://docs.python.org/c-api/init.html
>>> http://www.linuxjournal.com/article/3641

>>
>> CPython's GIL means that multithreading on multiple processors/cores has
>> limitations. Each interpreter has its own GIL, so processor-intensive
>> applications work better using the multiprocessing module than with the
>> threading module.

>
> I believe you will find the above is incorrect - even with multiple
> interpreter states you still have a single GIL.
>

not according to the docs.

http://docs.python.org/library/multiprocessing.html :

multiprocessing is a package that supports spawning processes using an
API similar to the threading module. The multiprocessing package
offers both local and remote concurrency, effectively side-stepping
the Global Interpreter Lock by using subprocesses instead of threads.
Due to this, the multiprocessing module allows the programmer to fully
leverage multiple processors on a given machine. It runs on both Unix
and Windows.
> Mark
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>

 
Reply With Quote
 
Mark Hammond
Guest
Posts: n/a
 
      09-07-2009
On 8/09/2009 9:16 AM, Grant Edwards wrote:
> On 2009-09-07, Mark Hammond<(E-Mail Removed)> wrote:
>
>>> CPython's GIL means that multithreading on multiple
>>> processors/cores has limitations. Each interpreter has its own
>>> GIL, so processor-intensive applications work better using the
>>> multiprocessing module than with the threading module.

>>
>> I believe you will find the above is incorrect - even with
>> multiple interpreter states you still have a single GIL.

>
> Please explain how multiple processes, each with a separate
> Python interpreter, share a single GIL.



Sorry, my mistake, I misread the original - using multiple Python
processes does indeed have a GIL per process. I was referring to the
'multiple interpreters in one process' feature of Python which is
largely deprecated, but if used, all 'interpreters' share the same GIL.

To clarify: in a single process there will only ever be one GIL, but in
multiple processes there most certainly will be multiple GILs.

Apologies for the confusion...

Cheers,

Mark

 
Reply With Quote
 
ganesh
Guest
Posts: n/a
 
      09-08-2009
My application is a TCP server having multiple client connectons. C++
PTHREADS are for each connected socket and the message received on the
socket is evaluated by python functions.
If I use only one process level python interpreter, then every thread
has to lock the GIL & so blocking the other threads from executing the
python code even if it is not the same python function that locking
thread is calling.

-- That's why I tried using python interpreters per thread. But that
also required GIL locking & so cannot be used.

-- I cannot use python threads inside the Pyton intrepreter because
then I will have to have some mechanism for communiaction between C++
Pthreads with these python threads.

I think there is no way that we can achieve this because of the GIL
being a process level state. Least I can do is have one python
interpreter initialized in main thread and lock the GIL in every
thread for python calls.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
how to implement per-thread timer in a multi-thread program? liu yang C Programming 4 07-28-2008 09:12 PM
Multiple independent Python interpreters in a C/C++ program? skip@pobox.com Python 9 04-13-2008 04:16 PM
Embedding: many interpreters OR one interpreter with many thread states ? adsheehan@eircom.net Python 3 06-10-2005 03:18 AM
Multiple Interpreters In a Single Thread bmatt Python 2 09-29-2004 11:07 AM
allocate TWO interpreters in a C program? Torsten Mohr Python 2 04-05-2004 08:28 PM



Advertisments