Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > Out Of Memory! - error

Reply
Thread Tools

Out Of Memory! - error

 
 
Zaphod Beeblebrox
Guest
Posts: n/a
 
      08-18-2006
Hello All;

Please forgive me if this is somewhere in the group but I am a little
clueless about this:

When running a Perl script:
_____________________________________
Characteristics of this binary (from libperl):
Compile-time options: PERL_MALLOC_WRAP USE_LARGE_FILES USE_PERLIO
Built under linux
Compiled at Aug 17 2006 09:52:18
@INC:
/usr/local/lib/perl5/5.8.8/i686-linux
/usr/local/lib/perl5/5.8.8
/usr/local/lib/perl5/site_perl/5.8.8/i686-linux
/usr/local/lib/perl5/site_perl/5.8.8
/usr/local/lib/perl5/site_perl
___________________________

I get an Out Of Memory! error.

Now, I am attempting to pull all the lines out of a really large file :

2162788 -rwx------ 1 xxxxxxx xxxxxxx 2212526628 2006-08-16 15:08
Data_20060816.log

Sample of data:

20060816093111,@@a,B,NT,1.98,1.97,853,,1.99,948,,1 .98,1.97,1.97,U@,Z,Z,5000,79400,,,,,,131033,,|,1.9 7,1155735051,2B
20060816093118,@@a,Q00,NT,1.97,853,,,1.99,897,,,Z, 0,0,R,0,378651,1155735060
20060816093128,@@a,Q00,NT,1.97,853,,,1.99,948,,,Z, 0,0,R,0,395594,1155735070
20060816093133,@@a,Q00,NT,1.97,853,,,1.99,902,,,Z, 0,0,R,0,403342,1155735075
20060816093133,@@a,Q00,NT,1.97,1393,,,1.99,902,,,Z ,0,0,R,0,404373,1155735075
20060816093134,@@a,B,NT,1.99,1.97,1393,,1.99,902,, 1.99,1.97,1.97,U@,Z,Z,4600,84000,,,,,,143314,,|,1. 97,1155735075,3B

I want to strip out all of the lines that have the 8th field blank (
actually , NULL not blank).

Using the perl debugger (perl -d <script> <file> ) it fails on putting
the file into an array:
Loading DB routines from perl5db.pl version 1.28
Editor support available.

Enter h or `h h' for help, or `man perldebug' for more help.

main:TradeQuoteCanBlankExch.pl:10):
10: $now_string = strftime "%a%b%e%H:%M:%S%Y", localtime;
DB<1> n
main:TradeQuoteCanBlankExch.pl:11):
11: my $symbol = $ARGV[0];#gets the file
DB<1> n
main:TradeQuoteCanBlankExch.pl:12):
12: chomp($symbol);#doesn't do **** for some reason
DB<1> n
main:TradeQuoteCanBlankExch.pl:13):
13: open(MYFILE,$symbol) || die "opening testfile: $!";
DB<1> n
main:TradeQuoteCanBlankExch.pl:14):
14: @stuff=<MYFILE>;
DB<1> n

Sits here forever and then

Out of Memory!

This is the root of the problem. I am wondering if there is a more
efficient way of reading a really large file into an array.

Any words of wisdom?

Thanks in advance.

Z

 
Reply With Quote
 
 
 
 
usenet@DavidFilmer.com
Guest
Posts: n/a
 
      08-18-2006
Zaphod Beeblebrox wrote:

> I am wondering if there is a more efficient way of reading
> a really large file into an array.


The best way to do that is to not do that. It's rarely necessary or
advisable to read a file into an array (especially a large file).

Instead of doing something like this:

my @stuff=<MYFILE>;
foreach my $line_of_stuff( @stuff ) {
# do stuff with stuff
}

do something like this:

while ( my $line_of_stuff = <MYFILE> ) {
# do stuff with stuff
}

Except use lexical filehandles (with better names) and the
three-argument form of file open... but that's all another rant (read
more in 'Perl Best Practices' by Damian Conway)

--
David Filmer (http://DavidFilmer.com)

 
Reply With Quote
 
 
 
 
Zaphod Beeblebrox
Guest
Posts: n/a
 
      08-22-2006
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> Zaphod Beeblebrox wrote:
>
> > I am wondering if there is a more efficient way of reading
> > a really large file into an array.

>
> The best way to do that is to not do that. It's rarely necessary or
> advisable to read a file into an array (especially a large file).
>
> Instead of doing something like this:
>
> my @stuff=<MYFILE>;
> foreach my $line_of_stuff( @stuff ) {
> # do stuff with stuff
> }
>
> do something like this:
>
> while ( my $line_of_stuff = <MYFILE> ) {
> # do stuff with stuff
> }
>
> Except use lexical filehandles (with better names) and the
> three-argument form of file open... but that's all another rant (read
> more in 'Perl Best Practices' by Damian Conway)
>
> --
> David Filmer (http://DavidFilmer.com)



HEY!! Thanks so much!

I think you saved me a ton on money on asprin!!

That did the trick! Like you stated, why do I/O when you don't have to!

Cheers..

G

 
Reply With Quote
 
usenet@DavidFilmer.com
Guest
Posts: n/a
 
      08-22-2006
Zaphod Beeblebrox wrote:

> I think you saved me a ton on money on asprin!!


I was serious in recommending the book, Perl Best Practices - it should
be the second Perl book that anyone ever buys (I REALLY wish it had
been in print years ago when I was new to Perl). Use your unspent
asprin budget to grab a copy - you won't regret it.

> That did the trick! Like you stated, why do I/O when you don't have to!


You're still doing I/O (well, I at least), but you're only groking one
line at a time. Slurping an entire file into an array is a common
mistake, especially among newer programmers. There are times when you
need to do that, but usually it's not a good idea because you're
usually processing a file one line at a time.

--
David Filmer (http://DavidFilmer.com)

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Error ::: *** glibc detected *** ./a.out: double free or corruption (out): Pallav singh C++ 1 01-29-2012 12:15 PM
Index was out of range ERROR <NEWBIE ERROR> =?Utf-8?B?UnVkeQ==?= ASP .Net 4 07-06-2005 05:35 PM
DNS error or Time Out error following Broadband Upgrade =?Utf-8?B?RGF2aWQgRGF5?= Microsoft Certification 2 05-06-2005 06:33 AM
PIX 515 Can Ping Out-to-In but not any webservices getting out-to-in Johnny Davis Cisco 6 01-19-2004 05:33 PM
How do you figure out the LDAP://? ("Error authenticating. Error authenticating user. The specified domain either does not exist or could not be contacted") mrwoopey ASP .Net 3 06-30-2003 10:11 PM



Advertisments