Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > python performance on Solaris

Reply
Thread Tools

python performance on Solaris

 
 
inaf
Guest
Posts: n/a
 
      10-09-2009
I have been following this group for quite some time and I figured
(after searching enough on google --and on this group-- and not
finding anything useful) I could pose this question here. Can anyone
shed some light on python's performance on Solaris? My code seem to
return lookups from a in memory data structure I build combining bunch
of dictionaries and lists 6-8 times faster on a 32 bit Linux box than
on a Solaris zone. Is there anything that needs to be done
specifically when installing/compiling python to improve performance
or is it a known thing that python does not perform that well on
solaris?

Thanks..
 
Reply With Quote
 
 
 
 
Antoine Pitrou
Guest
Posts: n/a
 
      10-11-2009
inaf <cem.ezberci <at> gmail.com> writes:
>
> My code seem to
> return lookups from a in memory data structure I build combining bunch
> of dictionaries and lists 6-8 times faster on a 32 bit Linux box than
> on a Solaris zone.


Well, if your workload is CPU-bound, the issue here is not really Solaris vs.
Linux but rather CPU power. You should try to run a generic (non-Python) CPU
benchmark (*) on both systems, perhaps this 6-8 factor is expected. If only
Python shows such a performance difference, on the other hand, perhaps you can
give us more precisions on those systems.

Regards

Antoine.


(*) for example one of the C programs on http://shootout.alioth.debian.org/u64/c.php


 
Reply With Quote
 
 
 
 
inaf
Guest
Posts: n/a
 
      10-12-2009
On Oct 11, 6:59*am, Antoine Pitrou <(E-Mail Removed)> wrote:
> inaf <cem.ezberci <at> gmail.com> writes:
>
>
>
> > My code seem to
> > return lookups from a in memory data structure I build combining bunch
> > of dictionaries and lists 6-8 times faster on a 32 bit Linux box than
> > on a Solaris zone.

>
> Well, if your workload is CPU-bound, the issue here is not really Solaris vs.
> Linux but rather CPU power. You should try to run a generic (non-Python) CPU
> benchmark (*) on both systems, perhaps this 6-8 factor is expected. If only
> Python shows such a performance difference, on the other hand, perhaps you can
> give us more precisions on those systems.
>
> Regards
>
> Antoine.
>
> (*) for example one of the C programs onhttp://shootout.alioth.debian.org/u64/c.php


Good point. I failed to compare the CPU power on these machines.. 32
bit linux box I have is 2666 Mhz vs the Solaris zone is 1415 Mhz.. I
guess that explains Thank you for the tip..
 
Reply With Quote
 
Antoine Pitrou
Guest
Posts: n/a
 
      10-14-2009
inaf <cem.ezberci <at> gmail.com> writes:
>
> Good point. I failed to compare the CPU power on these machines.. 32
> bit linux box I have is 2666 Mhz vs the Solaris zone is 1415 Mhz.. I
> guess that explains Thank you for the tip..


You have to compare not only CPU frequencies but the CPU models.
Recently Sun has been selling CPUs optimized for multi-threading (e.g. the
"UltraSPARC T2" or Niagara CPUs) which have, by design, very poor
single-threaded performance. If your Solaris zone uses such a CPU then a 6-8x
difference in single-threaded performance compared to a modern Intel or AMD CPU
is totally expected.

Regards

Antoine.


 
Reply With Quote
 
Jorgen Grahn
Guest
Posts: n/a
 
      10-14-2009
On Wed, 2009-10-14, Antoine Pitrou wrote:
> inaf <cem.ezberci <at> gmail.com> writes:
>>
>> Good point. I failed to compare the CPU power on these machines.. 32
>> bit linux box I have is 2666 Mhz vs the Solaris zone is 1415 Mhz.. I
>> guess that explains Thank you for the tip..

>
> You have to compare not only CPU frequencies but the CPU models.


Yes, at least that. Megahertz figures have been useless for decades,
except in advertising.

> Recently Sun has been selling CPUs optimized for multi-threading (e.g. the
> "UltraSPARC T2" or Niagara CPUs) which have, by design, very poor
> single-threaded performance. If your Solaris zone uses such a CPU then a 6-8x
> difference in single-threaded performance compared to a modern Intel
> or AMD CPU
> is totally expected.


(Had to Google it. A "Solaris Zone" is apparently some kind of
virtualization thing, with low CPU overhead.)

s/multi-threading/multi-programming/ I suppose. I certainly hope you
can still get performance while running many separate true processes in
parallel.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
 
Reply With Quote
 
inaf
Guest
Posts: n/a
 
      10-14-2009
On Oct 14, 7:15*am, Antoine Pitrou <(E-Mail Removed)> wrote:
> inaf <cem.ezberci <at> gmail.com> writes:
>
>
>
> > Good point. I failed to compare the CPU power on these machines.. 32
> > bit linux box I have is 2666 Mhz vs the Solaris zone is 1415 Mhz.. I
> > guess that explains Thank you for the tip..

>
> You have to compare not only CPU frequencies but the CPU models.
> Recently Sun has been selling CPUs optimized for multi-threading (e.g. the
> "UltraSPARC T2" or Niagara CPUs) which have, by design, very poor
> single-threaded performance. If your Solaris zone uses such a CPU then a 6-8x
> difference in single-threaded performance compared to a modern Intel or AMD CPU
> is totally expected.
>
> Regards
>
> Antoine.


Antonie -- yes, you are right. Even the architecture of the two types
make a difference. I was under the impression that RISC based CPUs did
not need to have a very high clock speed and that they can perform
similarly compared to an x86 processor with higher clock speed. That
is why I was a bit surprised. I guess there could be other factors at
play. That's why I was asking if there are specific things to be done
while compiling Python on Solaris. I found some tips online which led
me to compile it with a different threading lib resulting in slightly
better performance after my original post.

In terms of the processors I have, please see below for details:

Status of virtual processor 40 as of: 10/14/2009 17:13:51
on-line since 07/23/2009 18:48:21.
The sparcv9 processor operates at 1415 MHz,
and has a sparcv9 floating point processor.

So I guess this is not one of those you are talking about..

Thanks..
 
Reply With Quote
 
John Nagle
Guest
Posts: n/a
 
      10-15-2009
inaf wrote:
> I have been following this group for quite some time and I figured
> (after searching enough on google --and on this group-- and not
> finding anything useful) I could pose this question here. Can anyone
> shed some light on python's performance on Solaris?


Note that multithreaded compute-bound Python programs really suck
on multiprocessors. Adding a second CPU makes the program go slower,
due to a lame mechanism for resolving conflicts over the global interpreter
lock.

John Nagle
 
Reply With Quote
 
Antoine Pitrou
Guest
Posts: n/a
 
      10-15-2009
Le Wed, 14 Oct 2009 22:39:14 -0700, John Nagle a écritÂ*:
>
> Note that multithreaded compute-bound Python programs really suck
> on multiprocessors. Adding a second CPU makes the program go slower,
> due to a lame mechanism for resolving conflicts over the global
> interpreter lock.


I'm not sure what you're talking about. Python has no "mechanism for
resolving conflicts over the global interpreter lock" (let alone a lame
one ), it just trusts the OS to schedule a thread only when it is not
waiting on an unavailable resource (a lock). The GIL is just an OS-level
synchronization primitive and its behaviour (fairness, performance) will
depend on the behaviour of the underlying OS.

If this belief is derived from Dave Beazley's slides (*) about Python's
multi-threading issues, I'd say some of his observations are rather
questionable. First because the measurements were done on OS X, which has
its own serious concurrency problems (**). Second because one of the
benchmarks is so synthetic that it doesn't reflect real-world use of
Python at all.

This is not to say there aren't any issues, of course.


(*) http://www.dabeaz.com/python/GIL.pdf

(**) http://www.nabble.com/Mutex-performance-td24892454.html

 
Reply With Quote
 
Aahz
Guest
Posts: n/a
 
      10-15-2009
In article <(E-Mail Removed)>,
inaf <(E-Mail Removed)> wrote:
>
>I have been following this group for quite some time and I figured
>(after searching enough on google --and on this group-- and not finding
>anything useful) I could pose this question here. Can anyone shed
>some light on python's performance on Solaris? My code seem to return
>lookups from a in memory data structure I build combining bunch of
>dictionaries and lists 6-8 times faster on a 32 bit Linux box than on a
>Solaris zone.


Also, assuming that a "zone" is some kind of virtual machine, it will
definitely impose its own overhead.
--
Aahz ((E-Mail Removed)) <*> http://www.pythoncraft.com/

"To me vi is Zen. To use vi is to practice zen. Every command is a
koan. Profound to the user, unintelligible to the uninitiated. You
discover truth everytime you use it." http://www.velocityreviews.com/forums/(E-Mail Removed)
 
Reply With Quote
 
Dieter Maurer
Guest
Posts: n/a
 
      10-17-2009
Antoine Pitrou <(E-Mail Removed)> writes on Thu, 15 Oct 2009 16:25:43 +0000 (UTC):
> Le Wed, 14 Oct 2009 22:39:14 -0700, John Nagle a écrit*:
> >
> > Note that multithreaded compute-bound Python programs really suck
> > on multiprocessors. Adding a second CPU makes the program go slower,
> > due to a lame mechanism for resolving conflicts over the global
> > interpreter lock.

>
> I'm not sure what you're talking about. Python has no "mechanism for
> resolving conflicts over the global interpreter lock" (let alone a lame
> one ), it just trusts the OS to schedule a thread only when it is not
> waiting on an unavailable resource (a lock). The GIL is just an OS-level
> synchronization primitive and its behaviour (fairness, performance) will
> depend on the behaviour of the underlying OS.


But, independent from the OS and the fairness/performance of the GIL
management itself: the GIL is there to prevent concurrent execution
of Python code. Thus, at any time, at most one thread (in a process)
is executing Python code -- other threads may run as well, as long
as they are inside non Python code but cannot be executing Python bytecode,
independent of available CPU resources. This implies that Python cannot
fully exploit the power of multiprocessors.

It is also true that adding CPUs may in fact reduce performance for
compute bound multithreaded Python programs. While the additional
computational resources cannot be use by Python, the additional overhead
(switching between CPUs) may reduce overall performance.
I agree with you that it is difficult to understand when this overhead
were really significant.

Dieter
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Performance Tutorials Services - Boosting Performance by DisablingUnnecessary Services on Windows XP Home Edition Software Engineer Javascript 0 06-10-2011 02:18 AM
Ruby on Solaris 10 performance problems Colin Mackenzie Ruby 15 01-30-2009 11:35 PM
java, multithreading, fork() - performance problem on Solaris bugbear Java 4 03-28-2008 07:25 PM
Web Form Performance Versus Single File Performance jm ASP .Net 1 12-12-2003 11:14 PM
JVM performance Linux versus AIX/Solaris thomhashi Java 2 10-31-2003 08:02 AM



Advertisments