Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Re: Shootout: the benchmarks game.

Reply
Thread Tools

Re: Shootout: the benchmarks game.

 
 
Isaac Gouy
Guest
Posts: n/a
 
      04-28-2008
On Apr 25, 12:57 pm, "Bo Persson" <(E-Mail Removed)> wrote:
> Isaac Gouy wrote:
> > On Apr 23, 1:38 pm, Razii <(E-Mail Removed)> wrote:
> >> On Wed, 23 Apr 2008 07:49:07 -0700 (PDT), Isaac Gouy

>
> >> <(E-Mail Removed)> wrote:
> >>> There is no benchmark selector on the main page

>
> >>>http://shootout.alioth.debian.org/

>
> >> There is. If you click on "Show" tab, it takes you to this page.

>
> >>http://shootout.alioth.debian.org/gp...?test=all?=all

>
> >> There is no -server -Xms64m on this ranking. That's how Bo
> >> apparently found the link. You should apologize by claiming Bo
> >> followed my link. He didn't. Also, you didn't explain why -Xms64m
> >> must always use -server? Why not -client sometimes if that helps
> >> in some cases? Or other flags?

>
> > Bo wrote: "I actually went to the main page, and chose binary-tree
> > in benchmark selector at the top of the page. That brought me
> > here...".

>
> >http://groups.google.com/group/comp....2a2ac332ddd04b

>
> > As you just discovered - Bo could not have chosen "binary-tree in
> > benchmark selector at the top of the page".

>
> > There is no "binary-tree in benchmark selector at the top of the
> > page".

>
> > There's a button hard-coded to an entirely different webpage!

>
> Ok, so I started from there (following Razii's link).
>
> I *could* have gone to the main page, pressed "show", and *then*
> selected binary-tree. Big deal!
>
> I now see that Java 6 -server is down to 17th place.
>
> Bo Persson




We don't use labels like 1st place, 2nd place, 3rd place on the
webpages because those labels don't say anything that's interesting.

We use labels that say how much slower the implementation is than the
fastest implementation.

Overall, the geometric mean of those ratios for the Java 6 -server
programs is 1.7 times larger than the geometric mean of those ratios
for the measured Intel C++ programs.
 
Reply With Quote
 
 
 
 
Isaac Gouy
Guest
Posts: n/a
 
      04-29-2008
On Apr 28, 1:33 pm, Razii <(E-Mail Removed)> wrote:
> On Fri, 25 Apr 2008 21:57:45 +0200, "Bo Persson" <(E-Mail Removed)> wrote:
> >I *could* have gone to the main page, pressed "show"...

>
> Exactly. And -Xms64m is not on that page.



And -Xms64m is not shown
- on the FAQ page
- on any of the Debian : AMDô Sempronô pages
- on numerous other pages


And -Xms64m is shown on the main page

"Java 6 -Xint ubiquitous bytecode interpreter virtual machine
Java 6 -Xint Debian
Java 6 -Xms64m initial java server heap size 64m
Java GNU gcj optimizing ahead-of-time compiler
Java GNU gcj Debian
JavaScript JavaScriptCore WebKit-based ECMA scripting implementation
Debian"

http://shootout.alioth.debian.org/



And -Xms64m is shown on this "Create your own Ranking" page

"1.6 Fortran Intel
1.7 Eiffel SmartEiffel
1.7 Java 6 -Xms64m
1.8 Java 6 -server
1.8 OCaml 2.33"

http://shootout.alioth.debian.org/gp...t=all&lang=all


And -Xms64m is shown on numerous other pages.




> >I now see that Java 6 -server is down to 17th place.

>
> The shootout site is bogus and should not be taken seriously. It
> forces java version to use only one flag in all tests, regardless if
> another flag would do better. The guy also posted bogus excuses why
> Jet is not on the site.
>
> Also, as someone pointed out, the newly added Lisaac numbers in
> binarytrees are bogus too.
>
> (left = NULL).if {
> left := TREENODE.clone;
> right := TREENODE.clone;
> };
>
> The above tests whether the node has a NULL left branch and only then
> creates two new fresh nodes for the left and right branches. Lisaac
> version does only a very small fraction of the allocations done by a
> correct implementation of the benchmark.


 
Reply With Quote
 
 
 
 
Isaac Gouy
Guest
Posts: n/a
 
      04-29-2008
On Apr 28, 1:33 pm, Razii <(E-Mail Removed)> wrote:
> On Fri, 25 Apr 2008 21:57:45 +0200, "Bo Persson" <(E-Mail Removed)> wrote:
> >I *could* have gone to the main page, pressed "show"...

>
> Exactly. And -Xms64m is not on that page.
>
> >I now see that Java 6 -server is down to 17th place.

>
> The shootout site is bogus and should not be taken seriously. It
> forces java version to use only one flag in all tests, regardless if
> another flag would do better. The guy also posted bogus excuses why
> Jet is not on the site.
>
> Also, as someone pointed out, the newly added Lisaac numbers in
> binarytrees are bogus too.
>
> (left = NULL).if {
> left := TREENODE.clone;
> right := TREENODE.clone;
> };
>
> The above tests whether the node has a NULL left branch and only then
> creates two new fresh nodes for the left and right branches. Lisaac
> version does only a very small fraction of the allocations done by a
> correct implementation of the benchmark.



Yes, yesterday, Vesa pointed out that the Lisaac binary-trees program
wasn't doing the right thing, and today that Lisaac binary-trees
program has been removed.

The benchmarks game shows full program source and build & run logs so
that many eyes can see what's going on and cry foul if they think
something is wrong.

4 days ago Vesa commented on reddit that the Eiffel binary-trees
program wasn't doing the right thing, and that Eiffel program was
removed, and someone contributed a new Eiffel program and today that
new Eiffel program is shown on the website.

Find bugs; Fix bugs.
 
Reply With Quote
 
Isaac Gouy
Guest
Posts: n/a
 
      04-29-2008
On Apr 28, 1:33 pm, Razii <(E-Mail Removed)> wrote:
-snip-
> The shootout site is bogus and should not be taken seriously. It
> forces java version to use only one flag in all tests, regardless if
> another flag would do better.

-snip-


It's been at least a week since you were shown that both -server and -
Xms64m are used

/opt/sun-jdk-1.6.0.02/bin/java $JDKFLAGS -server -Xms64m -Xbatch -
classpath binarytrees.javaxx-2.javaxx_run binarytrees %A

/opt/sun-jdk-1.6.0.04/bin/java $JDKFLAGS -server -Xms64m -Xbatch -
classpath chameneosredux.javaxx-2.javaxx_run chameneosredux %A

/opt/sun-jdk-1.6.0.03/bin/java $JDKFLAGS -server -Xms64m -Xbatch -
classpath fannkuch.javaxx-2.javaxx_run fannkuch %A

/opt/sun-jdk-1.6.0.02/bin/java $JDKFLAGS -server -Xms64m -Xbatch -
classpath fasta.javaxx-2.javaxx_run fasta %A

/opt/sun-jdk-1.6.0.02/bin/java $JDKFLAGS -server -Xms64m -Xbatch -
classpath knucleotide.javaxx_run knucleotide %A

etc etc

http://shootout.alioth.debian.org/gp...x&lang2=javaxx
 
Reply With Quote
 
Isaac Gouy
Guest
Posts: n/a
 
      04-30-2008
On Apr 29, 6:34 pm, Razii <(E-Mail Removed)> wrote:
> On Tue, 29 Apr 2008 11:32:45 -0700 (PDT), Isaac Gouy
>
> <(E-Mail Removed)> wrote:
> >> Exactly. And -Xms64m is not on that page.

>
> >And -Xms64m is not shown

>
> It's not shown when you click on "show" from the main page. That all
> that matters. I don't care where else it's shown. Regardless, the
> -Xms64m always uses -server flag when it shouldn't have to when client
> is faster in some cases (like startup).



You've spent 10 days making untrue claims:

Apr 20, "The site forces the java version to either use only -server
or use only -Xms64m flag ... In this benchmark, combining -Xms64m and -
server produce best result ..."

http://groups.google.com/group/comp....7c32cfcefbdf49


Apr 21, "The only flag I see are -server -Xbatch -classpath"

http://groups.google.com/group/comp....7362cf9db02083


Apr 22, "There is no -Xms64m and -server linked from the main
page. ... You dug up a link but it's not on the main page."

http://groups.google.com/group/comp....c282fa0bbac488

etc etc


Now instead of admitting that your claims were wrong, you've changed
what you claim.

(Note: you seem confused about the purpose of "startup")


> There are other flags that
> might change the result if there is heavy GC activity. They are not
> used either.


Apr 20, "In this benchmark (since it's dependent on GC performance),
adding the
flag -XX:NewRatio=1 can further improve the result. This is not so
obvious with n = 16 but with n = 18, you can see the difference. "

And that benchmark is shown on the website N=16.


Why don't you go through all the Java programs measuring every
combination of JVM flag and report what you find?
 
Reply With Quote
 
Bo Persson
Guest
Posts: n/a
 
      05-01-2008
Razii wrote:
> On Wed, 30 Apr 2008 23:43:33 -0500, Razii <(E-Mail Removed)>
> wrote:
>
>> g++ -pipe -O3 -fomit-frame-pointer -mfpmath=sse -msse2 fasta.cpp -o
>> fasta.exe

>
> By the way, using the flags -mfpmath=sse -msse2 means the compiled
> version won't run on anything without an sse2 instruction set (that
> means no Athlon XP).
>
> That's the problem with compiled version that a JIT compiler doesn't
> have since it knows on what processor it's running.


That's just not true, because the JIT compiler cannot do any magic
here.

If it really can run fast enough on an Athlon XP, just compiler for
that. On anything faster, it will just fly.

> Removing these
> flags makes g++ version about 25% slower.


Is that fast enough? Otherwise skip that target and compile with SSE2.

With JIT it will still be 25% slower. What's the advantage?


Bo Persson


 
Reply With Quote
 
Bo Persson
Guest
Posts: n/a
 
      05-01-2008
Razii wrote:
> On Thu, 1 May 2008 15:34:18 +0200, "Bo Persson" <(E-Mail Removed)> wrote:
>
>> Is that fast enough? Otherwise skip that target and compile with
>> SSE2.

>
> If you are selling commercial software, why would you want to skip
> Athlon XP?


I would if it isn't fast enough to run the software. If it is fast
enough, everything else is as well.

> It's still widely used. JIT doesn't have that problem. It
> knows what processor it's running on.


If it runs well enough on an Athlon XP, compile for that. It will run
extremenly well on an overclocked Core 2 Quad.

>
>> With JIT it will still be 25% slower. What's the advantage?

>
> JIT is new and evolving. It will get faster. In any case, as shown
> in this benchmark, java VM was faster in this benchmark than the
> native java compiler.


It can't get faster on old hardware. If your program needs SSE2 and
the processor doesn't have it, how is the JIT compiler supposed to fix
that?


Bo Persson





 
Reply With Quote
 
Isaac Gouy
Guest
Posts: n/a
 
      05-01-2008
On Apr 30, 9:43 pm, Razii <(E-Mail Removed)> wrote:
> On Wed, 30 Apr 2008 08:00:38 -0700 (PDT), Isaac Gouy
>
> <(E-Mail Removed)> wrote:
> >You've spent 10 days making untrue claims:

>
> No, you are wrong. There is no -Xms when you click on show tab from
> the main page. You know that very well. Why waste time on quibbling?



Now instead of admitting that your claims were wrong, you've changed
what you claim.



> >Why don't you go through all...

>
> Let's continue. The only thing changed is my OS (Vista) and computer
> which is now 3 core phenom. Also, instead of > /dev/null on cygwin, I
> will use > \Null (since that seems to be 200 times faster on windows).
> benchmark: fasta
> java version:http://shootout.alioth.debian.org/gp...est=fasta‚Ć©=j...
> C++ versionhttp://shootout.alioth.debian.org/gp4/benchmark.php?test=fasta‚Ć©=g...
>
> As a great example of c++ portability, this doesn't even compile with
> my version of g++ (mingw), so I will use cygwin to compile. gcc
> version 3.4.4 (cygming special)
>
> g++ -pipe -O3 -fomit-frame-pointer -mfpmath=sse -msse2 fasta.cpp -o
> fasta.exe
>
> jc -inline+ fasta (jet)
>
> N 250,000 2,500,000 25,000,000
> g++ 0.166s 0.954s 8.884s
> java 0.297s 1.347s 11.699s (-Xms64m -server)
> jet 0.420s 1.701s 14.602s
>
> (note this guy uses -Xbatch flag everywhere. I have to see how that
> helps anywhere. It's just slightly slower with -Xbatch flag if
> anything).
>
> HotSpot VM is faster than native Jet compiler once again? That's three
> in a row. g++ is only slightly faster, not 50 or 100 times as some
> think.




The numbers you've created show more difference between the c++
program and the java program, show the c++ program as even faster,
than the measurements on the benchmarks game.

http://shootout.alioth.debian.org/gp...xx-2&p4=java-2
 
Reply With Quote
 
Bo Persson
Guest
Posts: n/a
 
      05-02-2008
Razii wrote:
> On Thu, 1 May 2008 18:47:35 +0200, "Bo Persson" <(E-Mail Removed)> wrote:
>
>> If it runs well enough on an Athlon XP, compile for that. It will
>> run extremenly well on an overclocked Core 2 Quad.

>
> It could have run 25% faster on Core2. It's running 25% slower on
> Core 2 Quad because the package was compiled to include Athlon XP.
>
> What's so hard to understand here??


That there is no need for it, except in benchmarks.

If some software can render a video correctly on an Athlon XP, there
is just no need for it to run 25% faster on a newer machine. Should
the film run faster?!

>
>> It can't get faster on old hardware. If your program needs SSE2 and
>> the processor doesn't have it, how is the JIT compiler supposed to
>> fix that?

>
> The JIT compiler fixed it by not penalizing Core 2 Quad just because
> Athlon XP doesn't have sse2. It runs on Core2 with sse2 enabled, and
> it runs on Athlon XP. In the case of pre-compiled version, it either
> has sse2 enabled for Core2 and doesn't run on Athlon Xp or it has
> sse2 disabled for both.
>
> What's so hard to understand here?


If the program isn't fast enough without SSE2, what does the JIT
compiler do on the old Athlon?

If it is fast enough without SSE2, why bother using it?


The benefit of a JIT compiler is that some programs can be made to run
on Windows, Linux, and OS X without recompilation. Not that it runs
faster on a Quad - everything does that anyway!


Bo Persson


 
Reply With Quote
 
peter koch
Guest
Posts: n/a
 
      05-03-2008
On 2 Maj, 21:04, Razii <(E-Mail Removed)> wrote:
> On Fri, 2 May 2008 18:35:34 +0200, "Bo Persson" <(E-Mail Removed)> wrote:
> >That there is no need for it, except in benchmarks.

>
> >If some software can render a video correctly on an Athlon XP, there
> >is just no need for it to run 25% faster on a newer machine. Should
> >the film run faster?!

>
> No, there could be a need for it say in a chess engine. The faster the
> chess engine, the stronger it plays. The stronger it plays, more chess
> enthusiasts are likely to buy it. Now, if you compile it for Athlon
> XP, Core2 would take the penalty because it would run 25% slower. If
> you compile it for Core2 by including sse2, it won't run on Athlon XP.


Well, for a chess engine it is going to be more or less the same. A
25% change in capacity is not going to change the playing level so
much. A better example would have been for some kind of server where
the processing is CPU-dominated. Here, 25% more capacity would mean
25% more clients possible. But when you write programs for a server,
you normally can allow yourself to write code for the newest
processors only.

/Peter
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Benchmarks, you know ya wanna post em ;) RObErT_RaTh Hardware 115 11-26-2009 03:14 PM
FEAR - real world gameplay benchmarks at bit-tech Silverstrand Front Page News 1 11-01-2005 02:22 PM
First hands-on benchmarks of notebook using GeForce 7800 Go Silverstrand Front Page News 5 09-29-2005 01:39 PM
Third Party Benchmarks for Web Server abhi Java 1 09-28-2005 03:28 PM
porting spec benchmarks to Java Vasanth Venkatachalam Java 1 11-29-2004 06:57 PM



Advertisments