Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > Split a very large log file

Reply
Thread Tools

Split a very large log file

 
 
Terry.Riegel@gmail.com
Guest
Posts: n/a
 
      02-02-2007
I have a 9gig log file, I am trying to do something with it. I tried
to run it through awstats and after about 8 hours I got a segmentation
fault.

I think if I could break it up into 2 files I could process one , then
process the other. I am not very good at PERL, but have to believe
this would be a 5-10 line program.

Any ideas?

Thanks,

Terry Riegel

 
Reply With Quote
 
 
 
 
J. Gleixner
Guest
Posts: n/a
 
      02-02-2007
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> I have a 9gig log file, I am trying to do something with it. I tried
> to run it through awstats and after about 8 hours I got a segmentation
> fault.
>
> I think if I could break it up into 2 files I could process one , then
> process the other. I am not very good at PERL, but have to believe
> this would be a 5-10 line program.


Yep.. why not post what you've tried.

Also, if you're on *nix? There are a bunch of
commands that will do it:

man split
man head
man tail
 
Reply With Quote
 
 
 
 
xhoster@gmail.com
Guest
Posts: n/a
 
      02-03-2007
(E-Mail Removed) wrote:
> I have a 9gig log file, I am trying to do something with it. I tried
> to run it through awstats and after about 8 hours I got a segmentation
> fault.
>
> I think if I could break it up into 2 files I could process one , then
> process the other.


I would guess that doing this will lead to the wrong answers. If awstats
is at all well written, then it wouldn't be loading all the data into
memory (which I'm guessing is what precipitated the seg fault) unless that
is what it needed to do to get the right answer.


> I am not very good at PERL, but have to believe
> this would be a 5-10 line program.


I'd use GNU "split" rather than using Perl at all, but anyway you just
need to look in $. to decide if you should reopen to a new output
file.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: Latest Nvidia Drivers WHQL certification but very very Large.. EMB NZ Computing 6 08-15-2008 11:18 AM
Program to split very large files by line count Curious Joe C++ 2 11-10-2006 03:35 AM
SOAP and very very large numbers bmm Ruby 0 04-18-2006 11:14 PM
Quick Book file access very very very slow Thomas Reed Computer Support 7 04-09-2004 08:09 PM
How do I: Split a large file on record and data (file = 3GB) seansan Perl Misc 6 01-05-2004 02:51 PM



Advertisments