Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > high performance/threaded applications in Python - your experiences?

Reply
Thread Tools

high performance/threaded applications in Python - your experiences?

 
 
Jay Loden
Guest
Posts: n/a
 
      06-23-2007
All,

In studying Python, I have predictably run across quite a bit of talk about the GIL and threading in Python. As my day job, I work with a (mostly Java) application that is heavily threaded. As such our application takes good advantage of multiple processors and we can often scale through simply adding processing power to a server.

I was hoping for some experiences that some of you on the list may have had in dealing with Python in a high performance and/or threaded environment. In essence, I'm wondering how big of a deal the GIL can be in a real-world scenario where you need to take advantage of multiple processor machines, thread pools, etc. How much does it get in the way (or not), and how difficult have you found it to architect applications for high performance? I have read a number of articles and opinions on whether or not the GIL is a good thing, and how it affects threaded performance on multiple processor machines, but what I haven't seen is experiences of people who have actually done it and reported back "it was a nightmare" or "it's no big deal"

Your thoughts and opinions are welcome, especially those with relevant experiences. Thanks!

-Jay

 
Reply With Quote
 
 
 
 
Ivan Voras
Guest
Posts: n/a
 
      06-23-2007
Jay Loden wrote:

> I was hoping for some experiences that some of you on the list may have had in dealing with Python in a high performance and/or threaded environment. In essence, I'm wondering how big of a deal the GIL can be in a real-world scenario where you need to take advantage of multiple processor machines, thread pools, etc. How much does it get in the way (or not), and how difficult have you found it to architect applications for high performance? I have read a number of articles and opinions on whether or not the GIL is a good thing, and how it affects threaded performance on multiple processor machines, but what I haven't seen is experiences of people who have actually done it and reported back "it was a nightmare" or "it's no big deal"


The theory: If your threads mostly do IO, you can get decent CPU usage
even with Python. If the threads are CPU-bound (e.g. you do a lot of
computational work), you'll effectively only make use of one processor.

In practice, I've noticed that Python applications don't scale very much
across CPUs even if they're doing mostly IO. I blame cache trashing or
similar effect caused by too many global synchronization events. I
didn't measure but the speedup may even be negative with large-ish
number of CPUs (>=4).

OTOH, if you can get by with using forking instead of threads (given
enough effort) you can achieve very good scaling.

--
(\__/)
(O.o)
(> < )

This is Bunny.
Copy Bunny into your signature to help him on his way to world domination!


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.4 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGfTGhldnAQVacBcgRAqZaAKCG866O3Lg5DYH/fl9/Ig4EwclmSwCfXw6W
99Is2/l13kbYq6P+IJBne+w=
=x2s3
-----END PGP SIGNATURE-----

 
Reply With Quote
 
 
 
 
Josiah Carlson
Guest
Posts: n/a
 
      06-24-2007
Ivan Voras wrote:
> Jay Loden wrote:
>
>> I was hoping for some experiences that some of you on the list may have had in dealing with Python in a high performance and/or threaded environment. In essence, I'm wondering how big of a deal the GIL can be in a real-world scenario where you need to take advantage of multiple processor machines, thread pools, etc. How much does it get in the way (or not), and how difficult have you found it to architect applications for high performance? I have read a number of articles and opinions on whether or not the GIL is a good thing, and how it affects threaded performance on multiple processor machines, but what I haven't seen is experiences of people who have actually done it and reported back "it was a nightmare" or "it's no big deal"

>
> The theory: If your threads mostly do IO, you can get decent CPU usage
> even with Python. If the threads are CPU-bound (e.g. you do a lot of
> computational work), you'll effectively only make use of one processor.
>
> In practice, I've noticed that Python applications don't scale very much
> across CPUs even if they're doing mostly IO. I blame cache trashing or
> similar effect caused by too many global synchronization events. I
> didn't measure but the speedup may even be negative with large-ish
> number of CPUs (>=4).
>
> OTOH, if you can get by with using forking instead of threads (given
> enough effort) you can achieve very good scaling.


Also, see the 'processing' package in the Python cheeseshop. It allows
you to use processes rather than threads with most of the same
abstractions. I hear it recently acquired the ability to pass file
handles between processes on the same machine

- Josiah
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Multiple applications/multiple web.configs: how to structure a big collection of (seemingly) nested web applications? ASP .Net 3 06-14-2007 09:25 AM
Ajax Applications (75 links) : Sites/applications featuring/using Ajax. www.web20developers.com HTML 0 10-06-2006 03:28 AM
Difference between Python CGI applications and Php applications praba kar Python 2 05-04-2005 06:49 PM
Mystery of Java applications blocking Windows XP applications Wolfgang Java 5 02-19-2004 02:45 PM
web applications versus. web portal applications John Davis ASP .Net 0 08-21-2003 12:11 AM



Advertisments