Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Development for dual core machine

Reply
Thread Tools

Development for dual core machine

 
 
Andy
Guest
Posts: n/a
 
      08-19-2007
Hi guys,

I'm sorry, I'm not sure this is the correct group to be asking this
kind of question...

I'm going to develop a software package that includes a web server
(and PHP code) , a database, and some Python code of course. I am
being strongly suggested to make it to work on a dual- or multi-core
computer, but I'm confused on how to take advantage of the multiple
CPUs.

>From what I read, I think that simply by making the package run in

several separate processes (web server, database server, Python
interpreter, etc.), and/or using multiple threads (which I will
anyway) the package should be able to use multiple CPUs.

Is my understanding correct? So actually, all I have to do is just
write my multi-threaded Python code as usual? And how is it decided
which process and which threads go to CPU 1, CPU 2, etc.? Perhaps the
BIOS?

Any advice greatly appreciated.
Andy

 
Reply With Quote
 
 
 
 
Bjoern Schliessmann
Guest
Posts: n/a
 
      08-19-2007
Andy wrote:

> I'm going to develop a software package that includes a web server
> (and PHP code) , a database, and some Python code of course. I am
> being strongly suggested to make it to work on a dual- or
> multi-core computer,


No problem. CPython will work on any dual core CPU.

> but I'm confused on how to take advantage of the multiple CPUs.


First: Use a web server that can make use of multiple cores.

Second: Use a data base that can make use of multiple cores.

Third, for using multiple cores from CPython: this is an FAQ, please
look at the mailing list archives. Whether you should optimize your
python application to use all cores strongly depends on what your
Python application actually does.

> From what I read, I think that simply by making the package run in
> several separate processes (web server, database server, Python
> interpreter, etc.), and/or using multiple threads (which I will
> anyway) the package should be able to use multiple CPUs.


CPython has the GIL (global interpreter lock). Please search for it
in the archives, it's been discussed exhaustingly.

> And how is it decided which process and which threads go to CPU 1,
> CPU 2, etc.? Perhaps the BIOS?


No, the kernel (i. e., Linux). The BIOS is completely out of this.

Regards,


Björn

--
BOFH excuse #388:

Bad user karma.

 
Reply With Quote
 
 
 
 
samwyse
Guest
Posts: n/a
 
      08-19-2007
Andy wrote:
> Hi guys,
>
> I'm sorry, I'm not sure this is the correct group to be asking this
> kind of question...
>
> I'm going to develop a software package that includes a web server
> (and PHP code) , a database, and some Python code of course. I am
> being strongly suggested to make it to work on a dual- or multi-core
> computer, but I'm confused on how to take advantage of the multiple
> CPUs.
>
>>From what I read, I think that simply by making the package run in

> several separate processes (web server, database server, Python
> interpreter, etc.), and/or using multiple threads (which I will
> anyway) the package should be able to use multiple CPUs.
>
> Is my understanding correct? So actually, all I have to do is just
> write my multi-threaded Python code as usual? And how is it decided
> which process and which threads go to CPU 1, CPU 2, etc.? Perhaps the
> BIOS?
>
> Any advice greatly appreciated.
> Andy
>


The Python interpreter is not multi-cpu aware, so using Python threads
won't work on multiple CPUs. If your tasks are CPU-bound, then fork
multiple processes. Most web servers (Apache) can handle this
automatically for you.
 
Reply With Quote
 
Bryan Olson
Guest
Posts: n/a
 
      08-19-2007
Andy wrote:
> I'm going to develop a software package that includes a web server
> (and PHP code) , a database, and some Python code of course. I am
> being strongly suggested to make it to work on a dual- or multi-core
> computer, but I'm confused on how to take advantage of the multiple
> CPUs.
>
> From what I read, I think that simply by making the package run in
> several separate processes (web server, database server, Python
> interpreter, etc.), and/or using multiple threads (which I will
> anyway) the package should be able to use multiple CPUs.


Right. There is a theoretical possibility that Python's GIL
could limit thread parallelism, but in this kind of application
it should not matter in the least. Either you're I/O bound, or
there are plenty of tasks the operating system could schedule.

If you have the option to run the Python interpreter within a
web-server process, that's usually a performance winner.

> Is my understanding correct? So actually, all I have to do is just
> write my multi-threaded Python code as usual? And how is it decided
> which process and which threads go to CPU 1, CPU 2, etc.? Perhaps the
> BIOS?


The O.S. actually. A lot of really smart people have put a whole
lot of work into making the O.S. do that well.

If you usually write your Python apps multi-threaded, as I do,
that's fine. Multi-core efficiency has only a little to do with
it. Web service are, for the most part, intrinsically
parallelizable: run multiple web servers, and multiple Python
interpreters serving multiple connections.

The only hard part is shared data. Scalability is all about
the database.


--
--Bryan
 
Reply With Quote
 
Paul Rubin
Guest
Posts: n/a
 
      08-19-2007
Andy <(E-Mail Removed)> writes:
> >From what I read, I think that simply by making the package run in

> several separate processes (web server, database server, Python
> interpreter, etc.), and/or using multiple threads (which I will
> anyway) the package should be able to use multiple CPUs.


Python threading doesn't support multiple CPU's because of the GIL.

One approach to using your multiple cores is to embed a Python
interpreter into a web server that runs in multiple processes,
e.g. mod_python under Apache.

Another is write your application as a separate process (e.g. as an
FastCGI), then run multiple instances of it, connected to a concurrent
web server. For what I'm currently doing, we're using lighthttpd as
the http server and flup to connect to a set of Python FCGI's.

For that matter, if your machine is just dual core, maybe it's ok to
just run a single Python process, figuring that will run on one core
and your database/httpd will run on the other one. However, you
should figure the dual core situation won't last. Dual socket
motherboards are fairly cheap these days, so we have a number of
4-core machines (two AMD dual core cpu's); Intel is already shipping
quad core cpu's, so that puts 8 cores in a dual socket board (you can
already buy Macintoshes configured that way). Higher end server
motherboards have 4 sockets, so you have to expect that 16-core
servers will be common pretty soon.

So if you're working on a cpu-intensive application it's worth your
while figuring out how to parallelize it.

Note that the most careful concurrency stuff probably is in the
database. Serious ones already know how to use multiple CPU's.
 
Reply With Quote
 
Bjoern Schliessmann
Guest
Posts: n/a
 
      08-19-2007
samwyse wrote:
> The Python interpreter is not multi-cpu aware, so using Python
> threads won't work on multiple CPUs.


Are you sure about this?

Regards,


Björn

--
BOFH excuse #12:

dry joints on cable plug

 
Reply With Quote
 
Bjoern Schliessmann
Guest
Posts: n/a
 
      08-19-2007
Paul Rubin wrote:

> Python threading doesn't support multiple CPU's because of the
> GIL.


:s/support/take full advantage of/

Regards,


Björn

--
BOFH excuse #46:

waste water tank overflowed onto computer

 
Reply With Quote
 
Andy
Guest
Posts: n/a
 
      08-20-2007
Thanks guys for the suggestions.

Andy

 
Reply With Quote
 
Willi Richert
Guest
Posts: n/a
 
      08-20-2007
Andy wrote:

> Thanks guys for the suggestions.
>
> Andy


It might be that you have to set the CPU affinity for your python process so
that it works at all. I had that problem on a dual core machine with
hyperthread enabled. Using taskset
(http://www.linuxcommand.org/man_pages/taskset1.html) helped solving the
problem:

taskset -c <CPU#> python ...

Best regards,
wr


--
pinkrose.dhis.org,
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
2 Dual Core 3Ghz Xeon or 2 Quad Core 2.33Ghz Xeon? Adrian Windows 64bit 5 03-19-2007 07:40 PM
From single core to dual core =?Utf-8?B?Q2FybG9z?= Windows 64bit 26 08-06-2006 09:08 PM
Core Solo & Core Duo are not Core microarchitecture; 65nm Pentium M chips bigal Hardware 0 03-22-2006 11:24 AM
Dual Core Vs Single Core Processor Real World Performance Difference Edge Computer Information 3 03-15-2006 01:30 AM
posible: dual core + single core =?Utf-8?B?TmllbHMgQ2hyLg==?= Windows 64bit 7 11-22-2005 06:11 PM



Advertisments