Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > python/ruby benchmark.

Reply
Thread Tools

python/ruby benchmark.

 
 
Glenn Parker
Guest
Posts: n/a
 
      06-11-2005
Isaac Gouy wrote:
>
> As I said - There isn't a problem with benchmarks per-se; there can be
> a problem with how one chooses to interpret benchmarks.


That's a meaningless distinction. You are suggesting that people who
read your benchmarks will somehow interpret them and find information
that is simply not there. The benchmarks paint an overly simplified
picture, nothing more. The only way to interpret the numbers correctly
is to ignore them and look for more pragmatic examples.

I do think there is some small value in the comparitive coding samples,
but the rigidity of the rules means that there is very little ingenuity
allowed in the code, and thus all the code ends up looking extremely
similar, regardless of the language employed. If you enjoy probing
variations in whitespace and the placement of semi-colons, you will have
lots of fun.

> Faster to develop-in than what other languages - Smalltalk? Lisp?
> Mathematica?


Faster to develop in than C++, Perl, Java, Forth, Assembler, and indeed,
Lisp. God help you if you decide to implement a web framework or a
compiler in Mathematica.

The real problem with the alioth benchmarks is that they are run by
amateurs that allow themselves to be bound by rigid pseudo-academic
dogma, but they can't research and fix trivial problems on their own,
and then they whine when nobody does it for them. If you guys aren't
highly motivated to fix things, why do you think the rest of us will be?

I gave up on them a while ago.

--
Glenn Parker | glenn.parker-AT-comcast.net | <http://www.tetrafoil.com/>


 
Reply With Quote
 
 
 
 
Isaac Gouy
Guest
Posts: n/a
 
      06-11-2005


Glenn Parker wrote:
> Isaac Gouy wrote:
> >
> > As I said - There isn't a problem with benchmarks per-se; there can be
> > a problem with how one chooses to interpret benchmarks.

>
> That's a meaningless distinction. You are suggesting that people who
> read your benchmarks will somehow interpret them and find information
> that is simply not there. The benchmarks paint an overly simplified
> picture, nothing more. The only way to interpret the numbers correctly
> is to ignore them and look for more pragmatic examples.
>
> I do think there is some small value in the comparitive coding samples,
> but the rigidity of the rules means that there is very little ingenuity
> allowed in the code, and thus all the code ends up looking extremely
> similar, regardless of the language employed. If you enjoy probing
> variations in whitespace and the placement of semi-colons, you will have
> lots of fun.
>
> > Faster to develop-in than what other languages - Smalltalk? Lisp?
> > Mathematica?

>
> Faster to develop in than C++, Perl, Java, Forth, Assembler, and indeed,
> Lisp. God help you if you decide to implement a web framework or a
> compiler in Mathematica.


My mistake, I didn't realize Ruby was fixed to Rails, somewhere I got
the impression it was more general than that.


> The real problem with the alioth benchmarks is that they are run by
> amateurs that allow themselves to be bound by rigid pseudo-academic
> dogma, but they can't research and fix trivial problems on their own,
> and then they whine when nobody does it for them. If you guys aren't
> highly motivated to fix things, why do you think the rest of us will be?
>
> I gave up on them a while ago.


Who's whining? Those who wish to contribute will, those who don't won't.

 
Reply With Quote
 
 
 
 
Alexandru Popescu
Guest
Posts: n/a
 
      06-11-2005
I've run through this thread and i am finally writting this because i am in a way astonished.

Afaik Python uses bytecodes while Ruby is completely interpretative, so imo there is not possible to
drive conclusions from this benchmark.

A lot of _comparisons_ are made in this list (I see everywhere Ruby code has fewer lines than X,
Ruby dev is better than Y, etc); finally you may expect these other kinds of comparisons to take
place. As far as the 1st comparison was made, I see no benefit from hidding behind the finger and
disregarding from the start the possibility to look at other types of comparisons.
I see here (n.b. on the ML) lots of ideas and open minds while speaking of ruby (_and just ruby_)
and I appreciate this. But everytime, somebody/something is bringing into attention another
tool/language something bad/sad happens and all this openminded is vanishing away.

While looking at Python and Ruby as both being oo scripting languages, I agree I would like to
hear/see/read comparisons: from all possible points of view. And imo one of the most important is
the runtime performance. Maybe the development is easier and cheaper, but if the performance is bad
i will not do it. Also, even if I have to write fewer lines in X, but the tools for Y are offering
much more I will certanly go for Y (disclaimer: I am not speaking from the pov of writting a command
line parser utility or some other little toy that parse the log of my server, but from the pov of
developing a medium/large application).

A few months ago - not very many - I have started looking at Ruby. And I am enjoying its ideas a
lot. But, after some time, I am still not sure I will start working with it. I know, you may say:
`maybe Ruby is not for youŽ (and maybe, I can agree with this). I must sadly confess that I haven't
got the same impression while looking at Python (and from the sizes of communities I can deduce many
others haven't felt the same).

Finally, please consider the above not a violent complain, but a sad objection I'm having.

:alex |.::the_mindstorm::.|






 
Reply With Quote
 
Austin Ziegler
Guest
Posts: n/a
 
      06-11-2005
On 6/11/05, Alexandru Popescu <(E-Mail Removed)> wrote:
> I've run through this thread and i am finally writting this
> because i am in a way astonished.
>=20
> Afaik Python uses bytecodes while Ruby is completely
> interpretative, so imo there is not possible to drive conclusions
> from this benchmark.


Which is one of the many reasons that I have a problem with the
alioth shootout. Ultimately, it tries to treat compiled, semi-
compiled, and interpreted languages equally. They're not. Compiled
languages will be faster than semi-compiled (bytecode), which will
be faster than interpreted. The level of speed difference is a
matter of scale, and that scale also matters on development time and
clarity.

> A lot of _comparisons_ are made in this list (I see everywhere
> Ruby code has fewer lines than X, Ruby dev is better than Y, etc);
> finally you may expect these other kinds of comparisons to take
> place. As far as the 1st comparison was made, I see no benefit
> from hidding behind the finger and disregarding from the start the
> possibility to look at other types of comparisons. I see here
> (n.b. on the ML) lots of ideas and open minds while speaking of
> ruby (_and just ruby_) and I appreciate this. But everytime,
> somebody/something is bringing into attention another
> tool/language something bad/sad happens and all this openminded is
> vanishing away.


Sorry, but that's just not true. People *are* open-minded on this
list with respect to new ideas. On the other hand, many of us don't
want to import those ideas which we feel will change the fundamental
nature of Ruby. Most of us don't *quite* agree on what that
fundamental nature is, but hey, that's the nature of individuals

No one has said that Ruby can't afford to be faster, or that other
languages ARE faster. What we've said is that, in most cases, Ruby
is Fast Enough...and that it gives us pleasure to work with it and
we feel that we have a better grasp on our problem domain and that
we have a more powerful solution for less development time. All of
these items work together. You know what? Python developers
generally feel the same about their programs.

> While looking at Python and Ruby as both being oo scripting
> languages, I agree I would like to hear/see/read comparisons: from
> all possible points of view. And imo one of the most important is
> the runtime performance. Maybe the development is easier and
> cheaper, but if the performance is bad i will not do it.


See, the problem is that you can't really know what your performance
will be like until you've solved at least part of your problem
domain. The use of benchmarks -- especially brainless ones like
you'll find at the alioth shootout -- won't help you. Unless, of
course, your problem domain is writing benchmarks.

Look, there are people doing OpenGL work with Ruby. It's obviously
fast enough for *them*. ARPA uses Ruby to drive a large-scale Java
application. Rich Kilmer is working on making Ruby drive a Flash
application (what's the status on that, Rich?). Ara Howard uses Ruby
to do NOAA work on large weather pictures and to drive a cluster.
These are all things that require significant computing power. These
people have chosen to use Ruby. Why? Because it gives them pleasure,
and Ruby is Fast Enough in real world tests.

Will they complain if Ruby gets faster? Not at all. They'll be
ecstatic. But they're not complaining about Ruby's speed right now.

> Also, even if I have to write fewer lines in X, but the tools for
> Y are offering much more I will certanly go for Y (disclaimer: I
> am not speaking from the pov of writting a command line parser
> utility or some other little toy that parse the log of my server,
> but from the pov of developing a medium/large application).


IMO, this is a little short-sighted way to look at it. Sure,
VisualStudio gives you great tools to develop .NET applications.
*Damn* but does it ever give you great tools. (I've seen a preview
of what's coming in November. Damn, I say, Damn, that's nice.) But
you have to do a lot more with the languages behind VisualStudio to
get things done. In most cases, the Ruby program will be done or
deployable long before the .NET program (and *way* before a J2EE
program). Real world results will be measured. Real optimisations
can be performed.

> A few months ago - not very many - I have started looking at Ruby.
> And I am enjoying its ideas a lot. But, after some time, I am
> still not sure I will start working with it. I know, you may say:
> `maybe Ruby is not for you=B4 (and maybe, I can agree with this). I
> must sadly confess that I haven't got the same impression while
> looking at Python (and from the sizes of communities I can deduce
> many others haven't felt the same).


You know, I'm not sure whether Ruby is for you or not. Have you done
development in it? Has it shown itself to be Too Slow for your
problem domain(s)? Does it matter if Ruby finished the job one
second slower than Python? Five seconds? Five minutes? Ultimately,
it depends on the job as to how that can be answered.

Just as another note, I think that the RAA is pretty active. It's
run on WEBrick through an Apache proxy. Wow, huh? WEBrick is a
pretty performant web server written in Ruby.

-austin
--=20
Austin Ziegler * http://www.velocityreviews.com/forums/(E-Mail Removed)
* Alternate: (E-Mail Removed)


 
Reply With Quote
 
Isaac Gouy
Guest
Posts: n/a
 
      06-11-2005


Austin Ziegler wrote:
> On 6/11/05, Alexandru Popescu <(E-Mail Removed)> wrote:
> > I've run through this thread and i am finally writting this
> > because i am in a way astonished.
> >
> > Afaik Python uses bytecodes while Ruby is completely
> > interpretative, so imo there is not possible to drive conclusions
> > from this benchmark.

>
> Which is one of the many reasons that I have a problem with the
> alioth shootout. Ultimately, it tries to treat compiled, semi-
> compiled, and interpreted languages equally. They're not. Compiled
> languages will be faster than semi-compiled (bytecode), which will
> be faster than interpreted.


Saying "Compiled languages will be faster than..." treats them all
equally!

There are compiled, bytecode, interpreter, ... /implementations/ of
programming languages - there are interpreters for C as-well-as
compilers.

 
Reply With Quote
 
Phil Tomson
Guest
Posts: n/a
 
      06-12-2005
In article <(E-Mail Removed)>,
Austin Ziegler <(E-Mail Removed)> wrote:
>On 6/11/05, Alexandru Popescu <(E-Mail Removed)> wrote:
>> I've run through this thread and i am finally writting this
>> because i am in a way astonished.
>>=20
>> Afaik Python uses bytecodes while Ruby is completely
>> interpretative, so imo there is not possible to drive conclusions
>> from this benchmark.

>
>Which is one of the many reasons that I have a problem with the
>alioth shootout. Ultimately, it tries to treat compiled, semi-
>compiled, and interpreted languages equally. They're not. Compiled
>languages will be faster than semi-compiled (bytecode), which will
>be faster than interpreted. The level of speed difference is a
>matter of scale, and that scale also matters on development time and
>clarity.


I have to say that I tend to agree with the astonishment.

Here's what I mean:
Someone comes along and asks about python vs. Ruby performance and
references the alioth benchmarks. Several folks jump on the alioth
benchmarks and call them bogus. I see this as blaming the messanger.

The fact is that these benchmarks exist and they cover lots of different
types of algorithms. They're on the web and people will look at them -
no way to stop that from happening.

Someone mentioned the Ackermann benchmark so I had a look and found that
gawk did much better than Ruby - that's pathetic folks. Now, I know that
if you could do this in gawk it should be possible to do it in Ruby as well.
There aren't even mumbers for N=8 or N=9 for the Ruby version. Maybe it's
because the Ruby version was written poorly. Maybe it's because of some
issue within the Ruby interpreter itself - I don't know, I didn't really
take time to look into it just now. Either way, wouldn't it be valuable
to try to improve our score on that benchmark? If it's because it's
poorly written, then it's gravy. If it's an issue with the interpreter
then maybe it would be good to know about the issue for the YARV design.

Now sure you can say that the benchmarks are bogus (and who knows, maybe
they are) and then take all your marbles and go home, but the benchmarks
remain and other languages communities are trying to hone their numbers.
If we totally sit it out, well, that's not going to look too good.

Sure these benchmarks don't tell the whole story about how nice it is to
develop in Ruby vs. gawk, but performance _is_ an issue for some people
and they will use this set of benchmarks (or another) as a factor in
deciding which langauge to use. If Ruby consistently shows up down
towards the bottom of the list... well, it can give a false impression.

If we tell them that they really shouldn't worry about performance (and
who knows, maybe we would be right in doing so) it just sounds like so
much excuse-making.

I think that as a community we should be trying to compete in these
benchmarks just like other langauge communities are doing. Perhaps the
results can help us as we do move to a VM - if nothing else to show
that we're making improvements (look at them as a set of unit tests for
performance). We know that Ruby needs help in the
performance area. If we keep telling ourselves that Ruby is 'fast
enough' for our application (and it may well be) are we going to be
sitting still while other languages improve performance? Have we been
the hare for the last few years sitting around while the tortise (TCL,
for example, once considered a lot slower than Ruby) is passing us up
(by creating or improving their VMs, for example).

Of course we know that we can do all sorts of things to speed things up
by writing C extension code, for example - that's a given, and the fact
that it's so easy to do in Ruby makes it possible to get into a lot of
areas where you might normally need to consider a compiled langauge,
however, a lot of people out there want to see how Ruby performs when
you just write Ruby code (C is a compiled language, of course). In some
sense marrying Ruby and C might seem like cheating to them, I suppose (try
to see it from their perspective).

If it looks like some of the benchmarks are misleading or stacked against
us, maybe we can propose other benchmarks where Ruby shines (BTW: why
does it say that the Object Methods benchmark is being deprecated?),
after all, it appears to be an open source process.

Bottom line: Ruby needs help in the performance area. Let's admit that
and work on improving in that area instead of shooting the messenger -
it'll make us look like whiners if we keep doing the latter. I'd really
rather not have the Ruby community perceived that way from the outside.

In the meantime, perhaps we could have some Ruby quizzes that focus on
choosing a benchmark and optimizing it. I think we would be
better off as a community if we participate in the process rather than
boycott it.

....just my 2cents.

Phil
 
Reply With Quote
 
Alexandru Popescu
Guest
Posts: n/a
 
      06-12-2005
#: the mind was *winged* after Phil Tomson said on 6/12/2005 9:20 AM :#
> In article <(E-Mail Removed)>,
> Austin Ziegler <(E-Mail Removed)> wrote:
>>On 6/11/05, Alexandru Popescu <(E-Mail Removed)> wrote:
>>> I've run through this thread and i am finally writting this
>>> because i am in a way astonished.
>>>=20
>>> Afaik Python uses bytecodes while Ruby is completely
>>> interpretative, so imo there is not possible to drive conclusions
>>> from this benchmark.

>>
>>Which is one of the many reasons that I have a problem with the
>>alioth shootout. Ultimately, it tries to treat compiled, semi-
>>compiled, and interpreted languages equally. They're not. Compiled
>>languages will be faster than semi-compiled (bytecode), which will
>>be faster than interpreted. The level of speed difference is a
>>matter of scale, and that scale also matters on development time and
>>clarity.

>
> I have to say that I tend to agree with the astonishment.
>
> Here's what I mean:
> Someone comes along and asks about python vs. Ruby performance and
> references the alioth benchmarks. Several folks jump on the alioth
> benchmarks and call them bogus. I see this as blaming the messanger.
>
> The fact is that these benchmarks exist and they cover lots of different
> types of algorithms. They're on the web and people will look at them -
> no way to stop that from happening.
>
> Someone mentioned the Ackermann benchmark so I had a look and found that
> gawk did much better than Ruby - that's pathetic folks. Now, I know that
> if you could do this in gawk it should be possible to do it in Ruby as well.
> There aren't even mumbers for N=8 or N=9 for the Ruby version. Maybe it's
> because the Ruby version was written poorly. Maybe it's because of some
> issue within the Ruby interpreter itself - I don't know, I didn't really
> take time to look into it just now. Either way, wouldn't it be valuable
> to try to improve our score on that benchmark? If it's because it's
> poorly written, then it's gravy. If it's an issue with the interpreter
> then maybe it would be good to know about the issue for the YARV design.
>
> Now sure you can say that the benchmarks are bogus (and who knows, maybe
> they are) and then take all your marbles and go home, but the benchmarks
> remain and other languages communities are trying to hone their numbers.
> If we totally sit it out, well, that's not going to look too good.
>
> Sure these benchmarks don't tell the whole story about how nice it is to
> develop in Ruby vs. gawk, but performance _is_ an issue for some people
> and they will use this set of benchmarks (or another) as a factor in
> deciding which langauge to use. If Ruby consistently shows up down
> towards the bottom of the list... well, it can give a false impression.
>
> If we tell them that they really shouldn't worry about performance (and
> who knows, maybe we would be right in doing so) it just sounds like so
> much excuse-making.
>
> I think that as a community we should be trying to compete in these
> benchmarks just like other langauge communities are doing. Perhaps the
> results can help us as we do move to a VM - if nothing else to show
> that we're making improvements (look at them as a set of unit tests for
> performance). We know that Ruby needs help in the
> performance area. If we keep telling ourselves that Ruby is 'fast
> enough' for our application (and it may well be) are we going to be
> sitting still while other languages improve performance? Have we been
> the hare for the last few years sitting around while the tortise (TCL,
> for example, once considered a lot slower than Ruby) is passing us up
> (by creating or improving their VMs, for example).
>
> Of course we know that we can do all sorts of things to speed things up
> by writing C extension code, for example - that's a given, and the fact
> that it's so easy to do in Ruby makes it possible to get into a lot of
> areas where you might normally need to consider a compiled langauge,
> however, a lot of people out there want to see how Ruby performs when
> you just write Ruby code (C is a compiled language, of course). In some
> sense marrying Ruby and C might seem like cheating to them, I suppose (try
> to see it from their perspective).
>
> If it looks like some of the benchmarks are misleading or stacked against
> us, maybe we can propose other benchmarks where Ruby shines (BTW: why
> does it say that the Object Methods benchmark is being deprecated?),
> after all, it appears to be an open source process.
>
> Bottom line: Ruby needs help in the performance area. Let's admit that
> and work on improving in that area instead of shooting the messenger -
> it'll make us look like whiners if we keep doing the latter. I'd really
> rather not have the Ruby community perceived that way from the outside.
>
> In the meantime, perhaps we could have some Ruby quizzes that focus on
> choosing a benchmark and optimizing it. I think we would be
> better off as a community if we participate in the process rather than
> boycott it.
>
> ....just my 2cents.
>
> Phil
>
>


Lots of thanks Phil for better expressing my thoughts and feelings. That is exactly what I wanted to
say - unfortunately the feelings just cut off my vocabulary.

And just as a confirmation of my last paragraph:
[stupid-quoting-myself]
I know, you may say:
`maybe Ruby is not for youŽ (and maybe, I can agree with this). I must sadly confess that I haven't
got the same impression while looking at Python (and from the sizes of communities I can deduce many
others haven't felt the same).
[/stupid-quoting-myself]

I have received offline messages suggesting to go with other solutions and some solid arguments why
I should do it.

I have to agree that it is no pleasure to find your invention `hammeredŽ by a (possible wrong)
benchmark; but I think when this happens I would go out shouting my benchmarks where i am kicking ass.
I remember that Java community did this a lot while the first c++ vs java benchmarks have been
published.
I remember that a few years ago when such a benchmark (driven by `paidŽ money by MS), was proving
that .NET was surclassing J2EE, finally even IBM joined the war and proved it completely wrong.

:alex |.::the_mindstorm::.|


 
Reply With Quote
 
Stefan Lang
Guest
Posts: n/a
 
      06-12-2005
On Sunday 12 June 2005 09:20, Phil Tomson wrote:
[...]
> Bottom line: Ruby needs help in the performance area. =C2=A0Let's admit t=

hat
> and work on improving in that area instead of shooting the messenger -
> it'll make us look like whiners if we keep doing the latter. =C2=A0I'd re=

ally
> rather not have the Ruby community perceived that way from the outside.

[...]

Im slowly getting fed up with this whining, "Ruby is so slow".
Especially if it's compared to Python.

Ever directly compared Ruby vs. Python solutions with regards to
performance?
Python is damn slow at startup, despite loading byte- instead of
sourcecode. On my machine, there is (approximately) a linear
ratio between LOPC and startup time: About 0.1 second per 200
LOPC. I've a Ruby script and a Python script doing the same thing
with a roughly equal model: The Ruby version takes ~0.06s,
the Python script ~0.26s for startup. Just to emphasize that a
bit: that's a factor of 4.3. (And the Ruby script is even=20
longer because it has some additional features.)
=46or longer program runs, the Python version comes closer
(but never surpasses) the Ruby one.

In conclusion, IMO Ruby will perform better for batch processing
and I don't know if the opposite is true for bigger/longer running
applications. I don't believe that Python is faster for serious
apps (not just 5 line benchmarks) unless someone shows me a program
written in Ruby and Python were the Python one is significantly faster.

Stefan


 
Reply With Quote
 
Steven Jenkins
Guest
Posts: n/a
 
      06-12-2005
Phil Tomson wrote:
> Austin Ziegler <(E-Mail Removed)> wrote:
>>Which is one of the many reasons that I have a problem with the
>>alioth shootout. Ultimately, it tries to treat compiled, semi-
>>compiled, and interpreted languages equally. They're not. Compiled
>>languages will be faster than semi-compiled (bytecode), which will
>>be faster than interpreted. The level of speed difference is a
>>matter of scale, and that scale also matters on development time and
>>clarity.

>
> I have to say that I tend to agree with the astonishment.
>
> Here's what I mean:
> Someone comes along and asks about python vs. Ruby performance and
> references the alioth benchmarks. Several folks jump on the alioth
> benchmarks and call them bogus. I see this as blaming the messanger.


That's putting it politely. The argument against, which has merit, is
stated with so much hyperbole that it damages its own credibility.

First, there are the ad hominem attacks. Words like "brainless", "inane"
and "bullshit". Don't tell me these aren't personal; they are.

Then there's the rhetorical exaggeration: If this benchmark has flaws,
then it has no value. If this benchmark has no value, then no benchmarks
have value. Culminating, of course, in "Benchmarks, like statistics, are
lies." I suppose I could be charitable and interpret that as a
hyperbolic way to say "Benchmarks, like statistics, tell only part of
the story. Use caution in their construction and interpretation." If
that's what was meant, then the original statement is itself,
ironically, a "lie" in precisely the same way. If it was meant
literally, then it betrays a deep ignorance of the practice of science
and engineering, among other things.

In my line of work, we use benchmarks and statistics extensively. They
may be useful for lying; so what? They are far more powerful for telling
the truth.

Steve


 
Reply With Quote
 
Austin Ziegler
Guest
Posts: n/a
 
      06-12-2005
If you don't have time to see why Ruby did so poorly in the alioth
Ackermann function, why did you say anything? Sorry, but the alioth
benchmarks are crap -- and always will be crap. Trying to improve
Ruby's score in them is a loser's game.

-austin


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off




Advertisments