Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > using python interpreters per thread in C++ program

Reply
Thread Tools

using python interpreters per thread in C++ program

 
 
grbgooglefan
Guest
Posts: n/a
 
      09-07-2009
Hi

I've a multi-threaded C++ program, in which I want to use embedded
python interpreter for each thread. I am using Python 2.6.2 on Linux
for this.

When I tried to embed python interpreter per thread, I got crash when
the threads were calling Python's C APIs.

Can we not use python interpreters even private to each multiple
thread?

What is best way to embed python in multi-threaded C++ application?

Please guide.
 
Reply With Quote
 
 
 
 
sturlamolden
Guest
Posts: n/a
 
      09-07-2009
On 7 Sep, 07:17, grbgooglefan <(E-Mail Removed)> wrote:


> What is best way to embed python in multi-threaded C++ application?


Did you remeber to acquire the GIL? The GIL is global to the process
(hence the name).

void foobar(void)
{
PyGILState_STATE state = PyGILState_Ensure();

/* Safe to use Python C API here */

PyGILState_Release(state);
}


S.M.











> Please guide.


 
Reply With Quote
 
 
 
 
sturlamolden
Guest
Posts: n/a
 
      09-07-2009
On 7 Sep, 07:17, grbgooglefan <(E-Mail Removed)> wrote:

> Can we not use python interpreters even private to each multiple
> thread?


You can use multiple interpreters, but they share GIL. For example,
Python extension modules are DLLs and will be loaded only once for
each process - the OS makes sure of that. Since they are objects too,
there can only be one GIL per process.

The same goes for files: If an interpreter calls close on a file
handle, it is closed for the whole process, not just locally to the
interpreter. Global synchronization is therefore needed. (Maybe not
for your threaded app, but for any conceivable use of Python.)

If you need two isolated interpreters, you need to run them in
different processes.





 
Reply With Quote
 
ganesh
Guest
Posts: n/a
 
      09-07-2009
> Did you remeber to acquire the GIL? The GIL is global to the process

No, I did not use GIL.

-- Why do we need to use GIL even though python is private to each
thread?
-- For using GIL, do we need to initialize GIL at startup and destroy/
finalize it at end?
-- With GIL, we are not achieiving concurrent operations on python
interpreters across threads. When it comes to calling Py-C APIs,
program is working like in a single threaded mode. Is there any way,
we can avoid the GIL locking, etc?

Please guide. Thanks.
 
Reply With Quote
 
ganesh
Guest
Posts: n/a
 
      09-07-2009
> Did you remeber to acquire the GIL? The GIL is global to the process
> (hence the name).


No, I did not use GIL.
-- For using GIL, do we need to initialize GIL at startup and destroy/
finalize it at end?
-- Are there any configuration & build related flags that I need to
use to make this work?

Please guide. Thanks.
 
Reply With Quote
 
sturlamolden
Guest
Posts: n/a
 
      09-07-2009
On 7 Sep, 07:59, ganesh <(E-Mail Removed)> wrote:

> No, I did not use GIL.
> -- For using GIL, do we need to initialize GIL at startup and destroy/
> finalize it at end?
> -- Are there any configuration & build related flags that I need to
> use to make this work?
>
> Please guide. Thanks.


I just showed you how...



 
Reply With Quote
 
ganesh
Guest
Posts: n/a
 
      09-07-2009
On Sep 7, 2:04*pm, sturlamolden <(E-Mail Removed)> wrote:
> I just showed you how...


Modified the thread function to use these APIs, but the call to
PyGILState_Ensure() is not returning at all.

void *callPyFunction(void * arg)
{
// Method two to get function eval
long thridx=(long)arg;
printf("\n---->my num=%d, calling showstr pyfunction\n",thridx);
char callpyf[] = "showstr(100)\n";
PyGILState_STATE state = PyGILState_Ensure();
printf("after PyGILState_Ensure\n");
PyObject* pycall = PyRun_String(callpyf,Py_file_input,glb, loc);
if(pycall == NULL || PyErr_Occurred()){
printf("PyRun_String failed\n");
PyErr_Print();
} else
printf("%d thread called showstr pyfunction ok\n",thridx);
PyGILState_Release(state);
pthread_exit(NULL);
}
 
Reply With Quote
 
Graham Dumpleton
Guest
Posts: n/a
 
      09-07-2009
On Sep 7, 3:42*pm, sturlamolden <(E-Mail Removed)> wrote:
> On 7 Sep, 07:17, grbgooglefan <(E-Mail Removed)> wrote:
>
> > What is best way to embed python in multi-threaded C++ application?

>
> Did you remeber to acquire the GIL? The GIL is global to the process
> (hence the name).
>
> void foobar(void)
> {
> * * PyGILState_STATE state = PyGILState_Ensure();
>
> * * /* Safe to use Python C API here */
>
> * * PyGILState_Release(state);


You can't use that in this case as they specifically are talking about
a interpreter per thread, which implies creating additional sub
interpreters. The simplified GIL state API you mentioned only works
for threads operating in the main (first) interpreter created within
the process.

The OP can do what they want, but they need to user lower level
routines for creating their own thread state objects and acquiring the
GIL against them.

Graham
 
Reply With Quote
 
ganesh
Guest
Posts: n/a
 
      09-07-2009
On Sep 7, 3:41*pm, Graham Dumpleton <(E-Mail Removed)>
wrote:
> On Sep 7, 3:42*pm, sturlamolden <(E-Mail Removed)> wrote:
> interpreters. The simplified GIL state API you mentioned only works
> for threads operating in the main (first) interpreter created within
> the process.


I modified my program to have Py_Initialize and compilation of one
Python function done in main() thread. Then I am calling only that
function in callPyFunction() thread. But, this thread does not come
out of PyGILState_Ensure() function.

> The OP can do what they want, but they need to user lower level
> routines for creating their own thread state objects and acquiring the
> GIL against them.
>
> Graham


What are the "lower level routines" for creating own thread state
objects & acquiring GILs.
Also, where can I find more information about those routines?

Please guide. Thanks.
 
Reply With Quote
 
Graham Dumpleton
Guest
Posts: n/a
 
      09-07-2009
On Sep 7, 6:47*pm, ganesh <(E-Mail Removed)> wrote:
> On Sep 7, 3:41*pm, Graham Dumpleton <(E-Mail Removed)>
> wrote:
>
> > On Sep 7, 3:42*pm, sturlamolden <(E-Mail Removed)> wrote:
> > interpreters. The simplified GIL state API you mentioned only works
> > for threads operating in the main (first) interpreter created within
> > the process.

>
> I modified my program to have Py_Initialize and compilation of one
> Python function done in main() thread. Then I am calling only that
> function in callPyFunction() thread. But, this thread does not come
> out of PyGILState_Ensure() function.
>
> > The OP can do what they want, but they need to user lower level
> > routines for creating their own thread state objects and acquiring the
> > GIL against them.

>
> > Graham

>
> What are the "lower level routines" for creating own thread state
> objects & acquiring GILs.
> Also, where can I find more information about those routines?


Documentation is at:

http://docs.python.org/c-api/init.html

Are you really using sub interpreters though? There is no evidence of
that in the code you posted earlier.

Sure you just don't understand enough about it and so are using the
wrong terminology and all you really want to do is run externally
created threads through the one interpreter.

Using sub interpreters is not for the feint of heart and not something
you want to do unless you want to understand Python C API internals
very well.

Graham

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
how to implement per-thread timer in a multi-thread program? liu yang C Programming 4 07-28-2008 09:12 PM
Multiple independent Python interpreters in a C/C++ program? skip@pobox.com Python 9 04-13-2008 04:16 PM
Embedding: many interpreters OR one interpreter with many thread states ? adsheehan@eircom.net Python 3 06-10-2005 03:18 AM
Multiple Interpreters In a Single Thread bmatt Python 2 09-29-2004 11:07 AM
allocate TWO interpreters in a C program? Torsten Mohr Python 2 04-05-2004 08:28 PM



Advertisments