Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   Re: Can I beat perl at grep-like processing speed? (http://www.velocityreviews.com/forums/t397764-re-can-i-beat-perl-at-grep-like-processing-speed.html)

Tim Smith 12-29-2006 03:51 PM

Re: Can I beat perl at grep-like processing speed?
 


you may not be able to beat perl's regex speed, but you can take some steps to speed up your python program using map and filter

here's a modified python program that will do your search faster

#!/usr/bin/env python
import re
r = re.compile(r'destroy', re.IGNORECASE)

def stripit(x):
return x.rstrip("\r\n")

print "\n".join( map(stripit, filter(r.search, file('bigfile'))) )

#time comparison on my machine
real 0m0.218s
user 0m0.210s
sys 0m0.010s

real 0m0.464s
user 0m0.450s
sys 0m0.010s

#original time comparison on my machine

real 0m0.224s
user 0m0.220s
sys 0m0.010s

real 0m0.508s
user 0m0.510s
sys 0m0.000s

also, if you replace the regex with a test like lambda x: x.lower().find("destroy") != -1, you will get really close to the speed of perl's (its possible perl will even take this shortcut when getting such a simple regex

#here's the times when doing the search this way
real 0m0.221s
user 0m0.210s
sys 0m0.010s

real 0m0.277s
user 0m0.280s
sys 0m0.000s

-- Tim

-- On 12/29/06 "js " <ebgssth@gmail.com> wrote:

> Just my curiosity.
> Can python beats perl at speed of grep-like processing?
>
> $ wget http://www.gutenberg.org/files/7999/7999-h.zip
> $ unzip 7999-h.zip
> $ cd 7999-h
> $ cat *.htm > bigfile
> $ du -h bigfile
> du -h bigfile
> 8.2M bigfile
>
> ---------- grep.pl ----------
> #!/usr/local/bin/perl
> open(F, 'bigfile') or die;
>
> while(<F>) {
> s/[\n\r]+$//;
> print "$_\n" if m/destroy/oi;
> }
> ---------- END ----------
> ---------- grep.py ----------
> #!/usr/bin/env python
> import re
> r = re.compile(r'destroy', re.IGNORECASE)
>
> for s in file('bigfile'):
> if r.search(s): print s.rstrip("\r\n")
> ---------- END ----------
>
> $ time perl grep.pl > pl.out; time python grep.py > py.out
> real 0m0.168s
> user 0m0.149s
> sys 0m0.015s
>
> real 0m0.450s
> user 0m0.374s
> sys 0m0.068s
> # I used python2.5 and perl 5.8.6
> --
> http://mail.python.org/mailman/listinfo/python-list



Marc 'BlackJack' Rintsch 12-29-2006 04:15 PM

Re: Can I beat perl at grep-like processing speed?
 
In <mailman.2126.1167407451.32031.python-list@python.org>, Tim Smith
wrote:

> also, if you replace the regex with a test like lambda x:
> x.lower().find("destroy") != -1, you will get really close to the speed
> of perl's


Testing for ``'destroy' in x.lower()`` should even be a little bit faster.

Ciao,
Marc 'BlackJack' Rintsch


All times are GMT. The time now is 01:50 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.