Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > simple regex benchmark inaccuracies

Reply
Thread Tools

simple regex benchmark inaccuracies

 
 
it_says_BALLS_on_your forehead
Guest
Posts: n/a
 
      02-24-2006
these are the results:

smmk94<WSP1>$ try_regex2.pl
Benchmark: running greedy, stingy, each for at least 1 CPU seconds...
greedy: 0 wallclock secs ( 1.02 usr + 0.00 sys = 1.02 CPU) @
1235595.10/s (n=1260307)
stingy: 1 wallclock secs ( 1.09 usr + 0.00 sys = 1.09 CPU) @
1172432.11/s (n=1277951)
Rate stingy greedy
stingy 1172432/s -- -5%
greedy 1235595/s 5% --
smmk94<WSP1>$ try_regex2.pl
Benchmark: running greedy, stingy, each for at least 1 CPU seconds...
greedy: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @
1180321.70/s (n=1251141)
stingy: 1 wallclock secs ( 1.07 usr + 0.01 sys = 1.08 CPU) @
1261175.00/s (n=1362069)
Rate greedy stingy
greedy 1180322/s -- -6%
stingy 1261175/s 7% --
smmk94<WSP1>$ try_regex2.pl
Benchmark: running greedy, stingy, each for at least 1 CPU seconds...
greedy: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @
1180321.70/s (n=1251141)
stingy: 2 wallclock secs ( 1.05 usr + 0.00 sys = 1.05 CPU) @
1260306.67/s (n=1323322)
Rate greedy stingy
greedy 1180322/s -- -6%
stingy 1260307/s 7% --
smmk94<WSP1>$ try_regex2.pl
Benchmark: running greedy, stingy, each for at least 1 CPU seconds...
greedy: 1 wallclock secs ( 1.16 usr + 0.00 sys = 1.16 CPU) @
1248870.69/s (n=1448690)
stingy: 2 wallclock secs ( 1.10 usr + 0.00 sys = 1.10 CPU) @
1137400.91/s (n=1251141)
Rate stingy greedy
stingy 1137401/s -- -9%
greedy 1248871/s 10% --



and here is the code:
=====
#!/usr/local/bin/perl

use strict; use warnings;
use Benchmark qw(cmpthese);

sub alt {
my $input =
'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225 ';
return 1 if $input =~ /campaign\w*_fulltop\.gif/;
}

sub alt2 {
my $input =
'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225 ';
return 1 if $input =~ /campaign\w*?_fulltop\.gif/;
}

cmpthese(-1, {
greedy => \&alt,
stingy => \&alt2,
});



....any comments on why there's a flipflop?

 
Reply With Quote
 
 
 
 
xhoster@gmail.com
Guest
Posts: n/a
 
      02-24-2006
"it_says_BALLS_on_your forehead" <(E-Mail Removed)> wrote:
> these are the results:
>
> smmk94<WSP1>$ try_regex2.pl
> Benchmark: running greedy, stingy, each for at least 1 CPU seconds...
> greedy: 0 wallclock secs ( 1.02 usr + 0.00 sys = 1.02 CPU) @
> 1235595.10/s (n=1260307)
> stingy: 1 wallclock secs ( 1.09 usr + 0.00 sys = 1.09 CPU) @
> 1172432.11/s (n=1277951)
> Rate stingy greedy
> stingy 1172432/s -- -5%
> greedy 1235595/s 5% --


What version of Benchmark are you running?

....

>
> ...any comments on why there's a flipflop?


What is there to say? The difference between them is negligible, the
results of Benchmark are subject to random error, systematic error, and
correlation between errors--just like every other measurement is--and you
are not running them long enough to average out even the error components
that can be averaged out.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
 
Reply With Quote
 
 
 
 
it_says_BALLS_on_your forehead
Guest
Posts: n/a
 
      02-24-2006

http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:

>
> What version of Benchmark are you running?



$VERSION = 1.00;

with Perl 5.6.1

>
> ...
>
> >
> > ...any comments on why there's a flipflop?

>
> What is there to say? The difference between them is negligible, the
> results of Benchmark are subject to random error, systematic error, and
> correlation between errors--just like every other measurement is--and you
> are not running them long enough to average out even the error components
> that can be averaged out.


perhaps, although the disparity seems too large to attribute to those
errors. i could understand if the difference was within 5%, but in one
case, greedy was 10% faster than stingy, and in the other, stingy was
7% faster than greedy. i understand that there are many factors
involved, but still i had hoped that the results from Benchmark would
prove more reliable and consistent. FYI, i ran again with -5 as the
first param to cmpthese, and got these results:

smmk94<WSP1>$ try_regex2.pl
Benchmark: running greedy, stingy, each for at least 5 CPU seconds...
greedy: 28 wallclock secs ( 5.16 usr + -0.02 sys = 5.14 CPU) @
1219724.32/s (n=6269383)
stingy: 19 wallclock secs ( 5.09 usr + 0.01 sys = 5.10 CPU) @
1334927.45/s (n=6808130)
Rate greedy stingy
greedy 1219724/s -- -9%
stingy 1334927/s 9% --

 
Reply With Quote
 
ednotover@gmail.com
Guest
Posts: n/a
 
      02-24-2006
it_says_BALLS_on_your forehead wrote:

> these are the results:


> stingy 1172432/s -- -5%
> greedy 1235595/s 5% --


> greedy 1180322/s -- -6%
> stingy 1261175/s 7% --


> ...any comments on why there's a flipflop?


Probably because a large portion of what you're comparing is my $input
= 'blah'.

Ed


> sub alt {
> my $input =
> 'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225 ';
> return 1 if $input =~ /campaign\w*_fulltop\.gif/;
> }
>
> sub alt2 {
> my $input =
> 'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225 ';
> return 1 if $input =~ /campaign\w*?_fulltop\.gif/;
> }


 
Reply With Quote
 
it_says_BALLS_on_your forehead
Guest
Posts: n/a
 
      02-24-2006

(E-Mail Removed) wrote:
> it_says_BALLS_on_your forehead wrote:
>
> > these are the results:

>
> > stingy 1172432/s -- -5%
> > greedy 1235595/s 5% --

>
> > greedy 1180322/s -- -6%
> > stingy 1261175/s 7% --

>
> > ...any comments on why there's a flipflop?

>
> Probably because a large portion of what you're comparing is my $input
> = 'blah'.
>
> Ed


the input is a real data record, and since that is constant, would not
affect the results of the benchmark. Xho probably has the right of it,
although i still have my doubts. i don't think that the Benchmark
module can be completely absolved of responsibility for the
discrepancies in the results...

 
Reply With Quote
 
ednotover@gmail.com
Guest
Posts: n/a
 
      02-25-2006

it_says_BALLS_on_your forehead wrote:
> (E-Mail Removed) wrote:
> > it_says_BALLS_on_your forehead wrote:


> > > ...any comments on why there's a flipflop?

> >
> > Probably because a large portion of what you're comparing is my $input
> > = 'blah'.
> >
> > Ed

>
> the input is a real data record, and since that is constant, would not
> affect the results of the benchmark. Xho probably has the right of it,
> although i still have my doubts. i don't think that the Benchmark
> module can be completely absolved of responsibility for the
> discrepancies in the results...


If you try it, you'll see. If you run a comparison of your tested
function calls without the my declaration against those with the my
declaration, the functions run 60% faster. You can see below that the
RE resolutions are basically the same, whereas the exact same function
(with a my declaration) yields the type of flip flop you're seeing.
I'd assume the flip flop simply reflects the variances of memory
management (for handling the my declarations) on the machine at the
time of execution.

Ed


$ cat checkit.pl
#!/usr/local/bin/perl

use strict; use warnings;
use Benchmark qw(cmpthese);

sub my_call {
my $input =
'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225 ';
}

cmpthese(-5, {
once => \&my_call,
twice => \&my_call,
});

my $input =
'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225 ';

sub greedy {
return 1 if $input =~ /campaign\w*_fulltop\.gif/;
}

sub stingy {
return 1 if $input =~ /campaign\w*?_fulltop\.gif/;
}

cmpthese(-5, {
greedy => \&greedy,
stingy => \&stingy,
});
$ ./checkit.pl
Benchmark: running once, twice, each for at least 5 CPU seconds...
once: 6 wallclock secs ( 5.55 usr + 0.00 sys = 5.55 CPU) @
2945391.71/s (n=16346924)
twice: 5 wallclock secs ( 5.24 usr + 0.00 sys = 5.24 CPU) @
3180811.45/s (n=16667452)
Rate once twice
once 2945392/s -- -7%
twice 3180811/s 8% --
Benchmark: running greedy, stingy, each for at least 5 CPU seconds...
greedy: 5 wallclock secs ( 5.17 usr + 0.00 sys = 5.17 CPU) @
3359500.00/s (n=17368615)
stingy: 5 wallclock secs ( 5.23 usr + 0.00 sys = 5.23 CPU) @
3320958.89/s (n=17368615)
Rate stingy greedy
stingy 3320959/s -- -1%
greedy 3359500/s 1% --
$

 
Reply With Quote
 
it_says_BALLS_on_your_forehead
Guest
Posts: n/a
 
      02-25-2006

(E-Mail Removed) wrote:
> it_says_BALLS_on_your forehead wrote:
> > (E-Mail Removed) wrote:
> > > it_says_BALLS_on_your forehead wrote:

>
> > > > ...any comments on why there's a flipflop?
> > >
> > > Probably because a large portion of what you're comparing is my $input
> > > = 'blah'.
> > >
> > > Ed

> >
> > the input is a real data record, and since that is constant, would not
> > affect the results of the benchmark. Xho probably has the right of it,
> > although i still have my doubts. i don't think that the Benchmark
> > module can be completely absolved of responsibility for the
> > discrepancies in the results...

>
> If you try it, you'll see. If you run a comparison of your tested
> function calls without the my declaration against those with the my
> declaration, the functions run 60% faster. You can see below that the
> RE resolutions are basically the same, whereas the exact same function
> (with a my declaration) yields the type of flip flop you're seeing.
> I'd assume the flip flop simply reflects the variances of memory
> management (for handling the my declarations) on the machine at the
> time of execution.
>
> Ed
>
>
> $ cat checkit.pl
> #!/usr/local/bin/perl
>
> use strict; use warnings;
> use Benchmark qw(cmpthese);
>
> sub my_call {
> my $input =
> 'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225 ';
> }
>
> cmpthese(-5, {
> once => \&my_call,
> twice => \&my_call,
> });
>
> my $input =
> 'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225 ';
>
> sub greedy {
> return 1 if $input =~ /campaign\w*_fulltop\.gif/;
> }
>
> sub stingy {
> return 1 if $input =~ /campaign\w*?_fulltop\.gif/;
> }
>
> cmpthese(-5, {
> greedy => \&greedy,
> stingy => \&stingy,
> });
> $ ./checkit.pl
> Benchmark: running once, twice, each for at least 5 CPU seconds...
> once: 6 wallclock secs ( 5.55 usr + 0.00 sys = 5.55 CPU) @
> 2945391.71/s (n=16346924)
> twice: 5 wallclock secs ( 5.24 usr + 0.00 sys = 5.24 CPU) @
> 3180811.45/s (n=16667452)
> Rate once twice
> once 2945392/s -- -7%
> twice 3180811/s 8% --
> Benchmark: running greedy, stingy, each for at least 5 CPU seconds...
> greedy: 5 wallclock secs ( 5.17 usr + 0.00 sys = 5.17 CPU) @
> 3359500.00/s (n=17368615)
> stingy: 5 wallclock secs ( 5.23 usr + 0.00 sys = 5.23 CPU) @
> 3320958.89/s (n=17368615)
> Rate stingy greedy
> stingy 3320959/s -- -1%
> greedy 3359500/s 1% --
> $


that is very interesting. i based my method of benchmarking on Randal
Schwartz's, where he assigns the variable a value inside the
subroutine, so i thought that would be good:

http://www.stonehenge.com/merlyn/UnixReview/col49.html

....however, based on your results, maybe that's a bad idea?

 
Reply With Quote
 
ednotover@gmail.com
Guest
Posts: n/a
 
      03-04-2006
[ snipped discussion of how benchmarking functions with my declarations
can cause varying benchmark results due to varying execution times for
my() ]

it_says_BALLS_on_your_forehead wrote:

> that is very interesting. i based my method of benchmarking on Randal
> Schwartz's, where he assigns the variable a value inside the
> subroutine, so i thought that would be good:
>
> http://www.stonehenge.com/merlyn/UnixReview/col49.html
>
> ...however, based on your results, maybe that's a bad idea?


My rule of thumb is to benchmark exactly what I'm trying to compare -
nothing more, and nothing less. So if I wanted to compare regexes,
then that's all I'd put into the comparison (as discussed previously in
this thread). It's like if I want to compare driving times from
Chicago to LA versus Chicago to Seattle. One way would be comparing
driving times from Miami to Chicago to LA versus Miami to Chicago to
Seattle - but why include the duplicated Miami to Chicago leg in there?
Remove the common factors and let the desired comparison stand alone.

So, yes, (in this one instance) I disagree with what Randal did there.
Note, though, that his results are such that a five to ten percent
swing wouldn't make any real difference. In your case, that five to
ten percent swing was effectively the entire difference.

As an aside, part of the issue in your code is that what you are
comparing is so quick, the my declarations contribute a significant
portion of the run time. Were you comparing something that took a good
bit longer, then the my declarations would be a minimal fraction of the
run time, and the run time variances in my() would be trivial compared
to the rest of the compared code.

Ed

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
inaccuracies in compression algorithm cornelis van der bent C Programming 18 07-24-2009 04:35 PM
How make regex that means "contains regex#1 but NOT regex#2" ?? seberino@spawar.navy.mil Python 3 07-01-2008 03:06 PM
Microsoft Practice test inaccuracies Larry MCSA 1 08-14-2005 09:31 AM
Inaccuracies in absolute positioning Csaba2000 Javascript 2 09-02-2003 09:00 AM
A very simple benchmark of Python based web server: Zope, mod_python with Apache, Twisted Framework Bowen Chiu Python 1 08-09-2003 05:50 PM



Advertisments