Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Need some hints on speeding up

Reply
Thread Tools

Need some hints on speeding up

 
 
Spamtrap
Guest
Posts: n/a
 
      08-11-2004
I only do occasional Perl programming and most things I write are
short processes. I have something I'm working on that is scanning a
text file with about 15 million lines and trying to extract matches
from another text file, which has about 170 entries. The second text
file is read into an array. The process then scans through the big
file for certain possible patterns - it will find those in about 1 out
of 25 lines,, when it finds one, it then loops through the array
trying to find a match there, and then writes out a couple of lines
into another text file.

It also writes to the screen a summary line about every 25th record.

When I run this it takes anywhere from 1.5 to 7.5 hours. It seems that
to avoid the 7.5 hour time, I should fresh reboot, and ctrl+alt+del
almost everything.

But even 1.5 hours is too long since I need to be able to run this
with different sets of data several times a day,

One thought I had is that writting a summary progress to the screen
[which helps me judge how far along it is, may be slowing things down]

Running Windows 98 SE and the latest version of Active Perl.
 
Reply With Quote
 
 
 
 
Jim Gibson
Guest
Posts: n/a
 
      08-11-2004
In article <(E-Mail Removed)>, Spamtrap
<(E-Mail Removed)> wrote:

> I only do occasional Perl programming and most things I write are
> short processes. I have something I'm working on that is scanning a
> text file with about 15 million lines and trying to extract matches
> from another text file, which has about 170 entries. The second text
> file is read into an array. The process then scans through the big
> file for certain possible patterns - it will find those in about 1 out
> of 25 lines,, when it finds one, it then loops through the array
> trying to find a match there, and then writes out a couple of lines
> into another text file.
>
> It also writes to the screen a summary line about every 25th record.
>
> When I run this it takes anywhere from 1.5 to 7.5 hours. It seems that
> to avoid the 7.5 hour time, I should fresh reboot, and ctrl+alt+del
> almost everything.
>
> But even 1.5 hours is too long since I need to be able to run this
> with different sets of data several times a day,
>
> One thought I had is that writting a summary progress to the screen
> [which helps me judge how far along it is, may be slowing things down]
>
> Running Windows 98 SE and the latest version of Active Perl.


People are going to need a little bit more information to help you.

First of all, comp.lang.perl is a defunct newsgroup. You would do
better to post to comp.lang.perl.misc.

What kind of hardware are you using? Your program may be limited by CPU
speed, disk I/O speed, or memory size. Can you upgrade your hardware if
that proves to be the limiting factor?

How long does it take to read through the 15M-line file? That gives you
a baseline for the minimum amount of time it will take to process the
file. If your full program takes considerably more than that, you may
be using a slow search algorithm or have other problems. Printing will
show down your program somewhat, but probably not a significant amount
unless you are really printing too much. Cut down the amount and see.

The best thing you can do is post a minimal, complete program (to
comp.lang.perl.misc) that people can inspect. Normally, you would want
a runnable program, but since you can't post the big text file or even
the 170 line pattern file, you may have to rely on code inspection
rather than profiling. Be sure and make your posted program readable.

You might want to profile your program. Check out 'perldoc -q profile'.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
hints for find-as-you-type hints cbossens73@yahoo.fr Java 2 04-08-2009 11:08 AM
Need some help speeding up this loop erikcw Python 2 10-30-2008 10:01 AM
PyInstaller: Need some hints (perhaps a good example?) Thin Myrna Python 0 01-03-2008 07:51 AM
Some general hints on debugging Java (Swing) needed Gertraud Java 2 03-17-2005 06:59 AM
Need some hints before starting to change a page Corobori ASP .Net Datagrid Control 1 03-04-2005 02:59 AM



Advertisments