Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Threaded GUI slowing method execution?

Reply
Thread Tools

Re: Threaded GUI slowing method execution?

 
 
Dave Angel
Guest
Posts: n/a
 
      10-02-2009
Aaron Hoover wrote:
> <div class="moz-text-flowed" style="font-family: -moz-fixed">I have a
> wx GUI application that connects to a serial port in a separate
> thread, reads from the port, and then is supposed to put the data it
> finds into a queue to be used by the main GUI thread. Generally
> speaking, it's working as expected.
>
> However, one method (that's part of a library I've written to parse
> the packet structure of the data that's coming over the serial port)
> executes approximately 1000 times slower (50ms vs. 50us) when called
> from the serial management thread in the GUI as compared to calling
> the same function from within a command line Python script. I checked
> it by wrapping the call as follows in both cases (GUI and command
> line script):
>
> tic = time.time()
> <method call>
> print time.time() - tic
>
> All the thread is doing most of the time is sitting around checking
> the serial port for waiting data, reading it, and appending it to a
> list when it finds it. Then, in the same thread, the method that seems
> to be remarkably slow works its way through that list looking for
> packets of data and appending the packet payloads it finds to a queue
> to be handled in some way by the GUI.
>
> My question is, what am I missing about either threading or the fact
> that this is running in a GUI that might explain such a huge slowdown.
> I'm sending data over the serial at a true rate of about 24k bytes per
> second, or approximately 2 packets per ms. Is it too much to ask to be
> able to process this data in realtime from within a GUI (I'm not
> talking about plotting or anything - just read it and find packets)?
> Like I said, the process pretty much runs in realtime from a command
> line script.
>
> This packet parsing needs to happen continuously, so it seems calling
> join() to ensure it's not interrupted by the GUI thread, won't work.
>
> Thanks in advance for your help.
>
> Aaron
>
> </div>
>

Threading in any language is tricky, so it may be just that. But in
Python in particular, you have the GIL (Global Interpreter Lock, I
believe), which usually makes writing threaded code easier, but can
really hurt efficiency. If you do a search of this list for GIL,
you'll see lots of discussion. Or you can google for "GIL python".
Look also at http://www.dabeaz.com/python/GIL.pdf

I'm no expert here, but let me try to summarize what I recall. The GIL
makes writing background threads easier when all but one of the threads
are frequently blocked for system calls. But as soon as you have two
threads doing "busy work," instead of them getting 50% each, the
threading overhead goes way up, and you spend much of your time just
deciding what not to do next. And it gets enormously worse than that
when there are more than one CPU available (eg. multicore).

Many people have concluded that (in Python) much of what threads are
used for should be done with processes.

DaveA

 
Reply With Quote
 
 
 
 
sturlamolden
Guest
Posts: n/a
 
      10-02-2009
On 2 Okt, 13:29, Dave Angel <(E-Mail Removed)> wrote:

> Many people have concluded that (in Python) much of what threads are
> used for should be done with processes.


Remember that threads were invented long before multi-core CPUs were
common. Java had threads before the VM could support more than one
CPU. The early Linux kernel had threads despite of the BKL, etc.
Threads have more usage than concurrent CPU bound busy work.

If you need to program multi-cores for speed, consider that you incur
a 200x speed penalty from Python alone. If you are worried about
speed, chances are you are not using Python anyway.

If you still have "need for speed" on a multicore, you can use Cython
and release the GIL when appropriate. Then launch multiple Python
threads and be happy.

In C, C++ and Fortran, one would not use threads for multicore
programming at all, but OpenMP.

Using more than one process is always an option, i.e. os.fork if you
have it or multiprocessing if you don't. Processes don't share GIL.

S.M.











 
Reply With Quote
 
 
 
 
Ole Streicher
Guest
Posts: n/a
 
      10-02-2009
sturlamolden <(E-Mail Removed)> writes:
> On 2 Okt, 13:29, Dave Angel <(E-Mail Removed)> wrote:
> If you are worried about speed, chances are you are not using Python
> anyway.


I *do* worry about speed. And I use Python. Why not? There are powerful
libraries available.

> If you still have "need for speed" on a multicore, you can use Cython
> and release the GIL when appropriate. Then launch multiple Python
> threads and be happy.


Usually this is not an option: numpy is AFAIK not available for Cython,
neither is scipy (ofcourse).

Especially for numeric calculations, speed *matters*.

> Using more than one process is always an option, i.e. os.fork if you
> have it or multiprocessing if you don't. Processes don't share GIL.


Not if the threads/processes need to share lots of data. Interprocess
communication can be very expensive -- even more if one needs to share
Python objects. Also, the support of sharing python objects between
processes seems to me not well supported at least by the standard python
libs.

Ole
 
Reply With Quote
 
sturlamolden
Guest
Posts: n/a
 
      10-02-2009
On 2 Okt, 20:19, Ole Streicher <(E-Mail Removed)> wrote:

> I *do* worry about speed. And I use Python. Why not? There are powerful
> libraries available.


I do as well. But "powerful libraries" should release the GIL. Let me
rephrase that: I am not worried about speed in the part of my code
that uses Python.


> Usually this is not an option: numpy is AFAIK not available for Cython,
> neither is scipy (ofcourse).


Anything available to Python is available to Cython.

Cython even has a special syntax for working with NumPy arrays.


> > Using more than one process is always an option, i.e. os.fork if you
> > have it or multiprocessing if you don't. Processes don't share GIL.

>
> Not if the threads/processes need to share lots of data. Interprocess
> communication can be very expensive -- even more if one needs to share
> Python objects.


I have written NumPy arrays that uses named shared memory as buffer.
They are pickled by name (i.e. the buffer is not copied) and therefore
very efficient when used with multiprocessing.
http://folk.uio.no/sturlamo/python/s...feb13-2009.zip

IPC without shared memory is generally cheap. The overhead of a pipe
or a unix domain socket is little more than that of a memcpy. The
expensive part is serializing and deserializing the Python object.

Also consider that the majority of the world's supercomputers are
programmed with MPI, which uses processes and IPC instead of threads.
On clusters, threads are not even an option. On shared memory
machines, MPI tends to be more efficient than threads/OpenMP (there
are often issues with cache use and false sharing when using threads).

S.M.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Threaded GUI slowing method execution? Aaron Hoover Python 1 10-02-2009 06:22 PM
Simple threaded app breaks on openStream method jmail@hursttechnical.com Java 1 05-15-2007 02:00 AM
Mozilla 1.7.8 slowing Broadback Firefox 0 06-13-2005 06:00 PM
CEF slowing down FE to FE traffic WAState Cisco 12 03-03-2005 06:29 PM
1721 Router Slowing Internet Access petemcc Cisco 2 11-21-2003 12:19 PM



Advertisments