Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > using python interpreters per thread in C++ program

Reply
Thread Tools

using python interpreters per thread in C++ program

 
 
sturlamolden
Guest
Posts: n/a
 
      09-08-2009
On 8 Sep, 04:22, ganesh <(E-Mail Removed)> wrote:

> My application is a TCP server having multiple client connectons. C++
> PTHREADS are for each connected socket and the message received on the
> socket is evaluated by python functions.
> If I use only one process level python interpreter, then every thread
> has to lock the GIL & so blocking the other threads from executing the
> python code even if it is not the same python function that locking
> thread is calling.


Usually, TCP-servers are I/O bound. You can safely use a single Python
process for this. A function evaluating a request will hold the GIL
for a while (but not until it's done). But most other threads will be
blocked waiting for I/O. Thus, there will be little contention for the
GIL anyway, and it should not affect scalability much. Only when
multiple requests are processed simultaneously will threre be
contention for the GIL. You can create high-performance TCP servers in
plain Python using e.g. Twisted. If you are in the strange situation
that a TCP server is compute-bound, consider using multiple processes
(os.fork or multiprocessing).


> I think there is no way that we can achieve this because of the GIL
> being a process level state. Least I can do is have one python
> interpreter initialized in main thread and lock the GIL in every
> thread for python calls.


I think you will find out your server is indeed I/O bound, like 99.9%
or all other TCP servers on this planet. Try to embed a single
interpreter first. Use the simplified GIL API I showed you. Most
likely you will find that it suffice.

If you need something more scalable, associate each pthread with a
separate Python process - e.g. using a named pipe on Windows or Unix
domain socket on Linux.
















 
Reply With Quote
 
 
 
 
ganesh
Guest
Posts: n/a
 
      09-08-2009
On Sep 8, 2:46*pm, I V <(E-Mail Removed)> wrote:
> Do you have to use threads? If you use a process per connection, rather
> than a thread, each process will have its own GIL.


No, i cannot change from threads to processes for handling
connections. This will change the complete design of our application
which is not feasilbe for python evaluation of the strings.
 
Reply With Quote
 
 
 
 
sturlamolden
Guest
Posts: n/a
 
      09-08-2009
On 8 Sep, 08:46, I V <(E-Mail Removed)> wrote:

> Do you have to use threads? If you use a process per connection, rather
> than a thread, each process will have its own GIL.


If ganesh is using Linux or Unix (which pthreads indicate), fork() is
just as efficient as threads.

On Windows one would need to keep a farm of prespawned Python
processes, connected with pipes to the main server.




 
Reply With Quote
 
sturlamolden
Guest
Posts: n/a
 
      09-08-2009
On 8 Sep, 09:14, ganesh <(E-Mail Removed)> wrote:

> No, i cannot change from threads to processes for handling
> connections. This will change the complete design of our application
> which is not feasilbe for python evaluation of the strings.


So the problem is actually bad design?
 
Reply With Quote
 
Graham Dumpleton
Guest
Posts: n/a
 
      09-08-2009
On Sep 8, 9:28*am, Mark Hammond <(E-Mail Removed)> wrote:
>*I was referring to the
> 'multiple interpreters in one process' feature of Python which is
> largely deprecated, ...


Can you please point to where in the documentation for Python it says
that support for multiple interpreters in one process is 'largely
deprecated'.

I know that various people would like the feature to go away, but I
don't believe I have ever seen an official statement from Guido or
other person in a position to make one, state that the official view
was that the API was deprecated.

Even in Python 3.1 the documentation for the APIs seems to merely
state some of the limitations and that it is a hard problem, even
still saying that problem would be addressed in future versions.

Graham
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
how to implement per-thread timer in a multi-thread program? liu yang C Programming 4 07-28-2008 09:12 PM
Multiple independent Python interpreters in a C/C++ program? skip@pobox.com Python 9 04-13-2008 04:16 PM
Embedding: many interpreters OR one interpreter with many thread states ? adsheehan@eircom.net Python 3 06-10-2005 03:18 AM
Multiple Interpreters In a Single Thread bmatt Python 2 09-29-2004 11:07 AM
allocate TWO interpreters in a C program? Torsten Mohr Python 2 04-05-2004 08:28 PM



Advertisments