Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: basic thread question

Reply
Thread Tools

Re: basic thread question

 
 
Jan Kaliszewski
Guest
Posts: n/a
 
      08-18-2009
18-08-2009 o 22:10:15 Derek Martin <(E-Mail Removed)> wrote:

> I have some simple threaded code... If I run this
> with an arg of 1 (start one thread), it pegs one cpu, as I would
> expect. If I run it with an arg of 2 (start 2 threads), it uses both
> CPUs, but utilization of both is less than 50%. Can anyone explain
> why?
>
> I do not pretend it's impeccable code, and I'm not looking for a
> critiqe of the code per se, excepting the case where what I've written
> is actually *wrong*. I hacked this together in a couple of minutes,
> with the intent of pegging my CPUs. Performance with two threads is
> actually *worse* than with one, which is highly unintuitive. I can
> accomplish my goal very easily with bash, but I still want to
> understand what's going on here...
>
> The OS is Linux 2.6.24, on a Ubuntu base. Here's the code:


Python threads can't benefit from multiple processors (because of GIL,
see: http://docs.python.org/glossary.html...erpreter-lock).

'multiprocessing' module is what you need:

http://docs.python.org/library/multiprocessing.html

Cheers,
*j

--
Jan Kaliszewski (zuo) <(E-Mail Removed)>
 
Reply With Quote
 
 
 
 
John Nagle
Guest
Posts: n/a
 
      08-24-2009
Jan Kaliszewski wrote:
> 18-08-2009 o 22:10:15 Derek Martin <(E-Mail Removed)> wrote:
>
>> I have some simple threaded code... If I run this
>> with an arg of 1 (start one thread), it pegs one cpu, as I would
>> expect. If I run it with an arg of 2 (start 2 threads), it uses both
>> CPUs, but utilization of both is less than 50%. Can anyone explain
>> why?
>>
>> I do not pretend it's impeccable code, and I'm not looking for a
>> critiqe of the code per se, excepting the case where what I've written
>> is actually *wrong*. I hacked this together in a couple of minutes,
>> with the intent of pegging my CPUs. Performance with two threads is
>> actually *worse* than with one, which is highly unintuitive. I can
>> accomplish my goal very easily with bash, but I still want to
>> understand what's going on here...
>>
>> The OS is Linux 2.6.24, on a Ubuntu base. Here's the code:

>
> Python threads can't benefit from multiple processors (because of GIL,
> see: http://docs.python.org/glossary.html...erpreter-lock).


This is a CPython implementation restriction. It's not inherent in
the language.

Multiple threads make overall performance worse because Python's
approach to thread locking produces a large number of context switches.
The interpreter unlocks the "Global Interpreter Lock" every N interpreter
cycles and on any system call that can block, which, if there is a
thread waiting, causes a context switch.

Multiple Python processes can run concurrently, but each process
has a copy of the entire Python system, so the memory and cache footprints are
far larger than for multiple threads.

John Nagle
 
Reply With Quote
 
 
 
 
Dennis Lee Bieber
Guest
Posts: n/a
 
      08-24-2009
On Sun, 23 Aug 2009 22:14:17 -0700, John Nagle <(E-Mail Removed)>
declaimed the following in gmane.comp.python.general:

> Multiple Python processes can run concurrently, but each process
> has a copy of the entire Python system, so the memory and cache footprints are
> far larger than for multiple threads.
>

One would think a smart enough OS would be able to share the
executable (interpreter) code, and only create a new stack/heap
allocation for data.
--
Wulfraed Dennis Lee Bieber KD6MOG
http://www.velocityreviews.com/forums/(E-Mail Removed) HTTP://wlfraed.home.netcom.com/

 
Reply With Quote
 
Piet van Oostrum
Guest
Posts: n/a
 
      08-24-2009
>>>>> Dennis Lee Bieber <(E-Mail Removed)> (DLB) wrote:

>DLB> On Sun, 23 Aug 2009 22:14:17 -0700, John Nagle <(E-Mail Removed)>
>DLB> declaimed the following in gmane.comp.python.general:


>>> Multiple Python processes can run concurrently, but each process
>>> has a copy of the entire Python system, so the memory and cache footprints are
>>> far larger than for multiple threads.
>>>

>DLB> One would think a smart enough OS would be able to share the
>DLB> executable (interpreter) code, and only create a new stack/heap
>DLB> allocation for data.


Of course they do, but a significant portion of a Python system consists
of imported modules and these are data as far as the OS is concerned.
Only the modules written in C which are loaded as DLL's (shared libs)
and of course the interpreter executable will be shared.
--
Piet van Oostrum <(E-Mail Removed)>
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: (E-Mail Removed)
 
Reply With Quote
 
Dave Angel
Guest
Posts: n/a
 
      08-24-2009
Dennis Lee Bieber wrote:
> On Sun, 23 Aug 2009 22:14:17 -0700, John Nagle <(E-Mail Removed)>
> declaimed the following in gmane.comp.python.general:
>
>
>> Multiple Python processes can run concurrently, but each process
>> has a copy of the entire Python system, so the memory and cache footprints are
>> far larger than for multiple threads.
>>
>>

> One would think a smart enough OS would be able to share the
> executable (interpreter) code, and only create a new stack/heap
> allocation for data.
>

That's what fork is all about. (See os.fork(), available on most
Unix/Linux) The two processes start out sharing their state, and only
the things subsequently written need separate swap space.

In Windows (and probably Unix/Linux), the swapspace taken by the
executable and DLLs(shared libraries) is minimal. Each DLL may have a
"preferred location" and if that part of the address space is available,
it takes no swapspace at all, except for static variables, which are
usually allocated together. I don't know whether the standard build of
CPython (python.exe and the pyo libraries) uses such a linker option,
but I'd bet they do. It also speeds startup time.

On my system, a minimal python program uses about 50k of swapspace. But
I'm sure that goes way up with lots of imports.


DaveA
 
Reply With Quote
 
Piet van Oostrum
Guest
Posts: n/a
 
      08-24-2009
>>>>> Dave Angel <(E-Mail Removed)> (DA) wrote:

>DA> Dennis Lee Bieber wrote:
>>> On Sun, 23 Aug 2009 22:14:17 -0700, John Nagle <(E-Mail Removed)>
>>> declaimed the following in gmane.comp.python.general:
>>>
>>>
>>>> Multiple Python processes can run concurrently, but each process
>>>> has a copy of the entire Python system, so the memory and cache footprints are
>>>> far larger than for multiple threads.
>>>>
>>>>
>>> One would think a smart enough OS would be able to share the
>>> executable (interpreter) code, and only create a new stack/heap
>>> allocation for data.
>>>

>DA> That's what fork is all about. (See os.fork(), available on most
>DA> Unix/Linux) The two processes start out sharing their state, and only the
>DA> things subsequently written need separate swap space.


But os.fork() is not available on Windows. And I guess refcounts et al.
will soon destroy the sharing.
--
Piet van Oostrum <(E-Mail Removed)>
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: (E-Mail Removed)
 
Reply With Quote
 
sturlamolden
Guest
Posts: n/a
 
      08-24-2009
On 24 Aug, 13:21, Piet van Oostrum <(E-Mail Removed)> wrote:

> But os.fork() is not available on Windows. And I guess refcounts et al.
> will soon destroy the sharing.


Well, there is os.fork in Cygwin and SUA (SUA is the Unix subsytem in
Windows Vista Professional). Cygwin's fork is a bit sluggish.

Multiprocessing works on Windows and Linux alike.

Apart from that, how are you going to use threads? The GIL will not be
a problem if it can be released. Mostly, the GIL is a hypothetical
problem. It is only a problem for compute-bound code written in pure
Python. But very few use Python for that. However, if you do and can
afford the 200x speed penalty from using Python (instead of C, C++,
Fortran, Cython), you can just as well accept that only one CPU is
used.


Sturla Molden











 
Reply With Quote
 
Piet van Oostrum
Guest
Posts: n/a
 
      08-24-2009
>>>>> sturlamolden <(E-Mail Removed)> (s) wrote:

>s> On 24 Aug, 13:21, Piet van Oostrum <(E-Mail Removed)> wrote:
>>> But os.fork() is not available on Windows. And I guess refcounts et al.
>>> will soon destroy the sharing.


>s> Well, there is os.fork in Cygwin and SUA (SUA is the Unix subsytem in
>s> Windows Vista Professional). Cygwin's fork is a bit sluggish.


That's because it doesn't use copy-on-write. Thereby losing most of its
advantages. I don't know SUA, but I have vaguely heard about it.
--
Piet van Oostrum <(E-Mail Removed)>
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: (E-Mail Removed)
 
Reply With Quote
 
sturlamolden
Guest
Posts: n/a
 
      08-25-2009
On 25 Aug, 01:26, Piet van Oostrum <(E-Mail Removed)> wrote:

> That's because it doesn't use copy-on-write. Thereby losing most of its
> advantages. I don't know SUA, but I have vaguely heard about it.


SUA is a version of UNIX hidden inside Windows Vista and Windows 7
(except in Home and Home Premium), but very few seem to know of it.
SUA (Subsystem for Unix based Applications) is formerly known as
Interix, which is a certified version of UNIX based on OpenBSD. If you
go to http://www.interopsystems.com (a website run by Interop Systems
Inc., a company owned by Microsoft), you will find a lot of common
unix tools prebuilt for SUA, including Python 2.6.2.

The NT-kernel supports copy-on-write fork with a special system call
(ZwCreateProcess in ntdll.dll), which is what SUA's implementation of
fork() uses.
 
Reply With Quote
 
Piet van Oostrum
Guest
Posts: n/a
 
      08-25-2009
>>>>> sturlamolden <(E-Mail Removed)> (s) wrote:

>s> On 25 Aug, 01:26, Piet van Oostrum <(E-Mail Removed)> wrote:
>>> That's because it doesn't use copy-on-write. Thereby losing most of its
>>> advantages. I don't know SUA, but I have vaguely heard about it.


>s> SUA is a version of UNIX hidden inside Windows Vista and Windows 7
>s> (except in Home and Home Premium), but very few seem to know of it.
>s> SUA (Subsystem for Unix based Applications) is formerly known as
>s> Interix, which is a certified version of UNIX based on OpenBSD. If you
>s> go to http://www.interopsystems.com (a website run by Interop Systems
>s> Inc., a company owned by Microsoft), you will find a lot of common
>s> unix tools prebuilt for SUA, including Python 2.6.2.


>s> The NT-kernel supports copy-on-write fork with a special system call
>s> (ZwCreateProcess in ntdll.dll), which is what SUA's implementation of
>s> fork() uses.


I have heard about that also, but is there a Python implementation that
uses this? (Just curious, I am not using Windows.)
--
Piet van Oostrum <(E-Mail Removed)>
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: (E-Mail Removed)
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
basic thread question Derek Martin Python 2 08-24-2009 02:20 PM
Re: basic thread question Derek Martin Python 2 08-20-2009 04:04 AM
Basic SocketServer & Thread Question john Java 4 06-03-2005 05:03 PM
PyNew_Interpreter(): "Fatal Python error, invalid thread state for this thread" question vincent wehren Python 0 12-11-2003 08:09 PM
Upgrading Microsoft Visual Basic 6.0 to Microsoft Visual Basic .NET Jaime MCSD 2 09-20-2003 05:16 AM



Advertisments