Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > perl eating up memory slowly until program stops - any ideas ?

Reply
Thread Tools

perl eating up memory slowly until program stops - any ideas ?

 
 
Jasper
Guest
Posts: n/a
 
      07-06-2004
Hello, I am clearing the contents of the arrays with @array = (); for
all arrays, and I print out the $#array to get the size while the
program is running for all arrays...and the arrays are not growing..so
I am stumped as to what could be causing the fact that memory needs
continue to increase...I am using GMP::BigFloat and doing large
precision subtractions..

Any tips are appreciated as to how I can manage/stop this..
Thank you,

Jasper
 
Reply With Quote
 
 
 
 
Matt Garrish
Guest
Posts: n/a
 
      07-06-2004

"Purl Gurl" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> Purl Gurl wrote:
>
> > Jasper wrote:

>
> (snipped)
>
> > > I am clearing the contents of the arrays with @array = (); for
> > > all arrays, and I print out the $#array to get the size while the
> > > program is running for all arrays...and the arrays are not growing..so
> > > I am stumped as to what could be causing the fact that memory needs
> > > continue to increase...I am using GMP::BigFloat and doing large
> > > precision subtractions..

>
> > Try lexical scoping for your arrays, within private blocks,
> > to release memory for reuse.

>
>
> However, what none will tell you is although lexical scoping
> releases memory for reuse, you will trade some efficiency for
> better memory management.
>
> Use of lexical scoping will decrease efficiency by percentages
> which are easy to measure.
>
> When memory is an issue, lexical scoping is useful, otherwise
> a programmer should question if lexical scoping is beneficial
> within a context of efficiency.
>
>
> Purl Gurl
> --
> #!perl
>
> print "Content-type: text/plain\n\n";
>
> use Benchmark;
>
> print "Run One:\n\n";
> &Time;
>
> print "\n\nRun Two:\n\n";
> &Time;
>
> print "\n\nRun Three:\n\n";
> &Time;
>
> sub Time
> {
> timethese (100000,
> {
> 'Purl_Gurl' =>
> '@Array1 = qw (Now is the time for all good men to come to the aid of

their country);
> @Array2 = qw (Behind every successful man walks a woman pushing him

along);
> $var1 = "@Array1";
> @Array1 = ();
> $var2 = "@Array2";
> @Array2 = ();',
>
> 'Purl_Gurl2' =>
> '{
> my @Array3 = qw (Now is the time for all good men to come to the aid

of their country);
> my $var3 = "@Array3";
> @Array3= ();
> }
> {
> my @Array4 = qw (Behind every successful man walks a woman pushing

him along);
> my $var4 = "@Array4";
> @Array4= ();
> }',
> } );
> }
>
>


If you're trying to prove that well-conceived and structured code might
sometimes run marginally slower than spaghetti code, you're not exactly onto
something new (or do you just randomly wrap code in blocks for fun). The
fact remains, however, that spaghetti code is usually a sign of bad code,
and optimizing it will more often than not reduce the runtime anyway. But I
suppose there's no point in preaching to someone who has never had a job as
a programmer. Rulez are for losers! Rock on PG!

Matt


 
Reply With Quote
 
 
 
 
Matt Garrish
Guest
Posts: n/a
 
      07-07-2004

"Purl Gurl" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> Matt Garrish continues to ignorantly troll:
>
> > Purl Gurl wrote:
> > > Purl Gurl wrote:
> > > > Jasper wrote:

>
> (snipped)
>
> > > > Try lexical scoping for your arrays, within private blocks,
> > > > to release memory for reuse.

>
> > > However, what none will tell you is although lexical scoping
> > > releases memory for reuse, you will trade some efficiency for
> > > better memory management.

>
>
> > If you're trying to prove that well-conceived and structured code might
> > sometimes run marginally slower than spaghetti code,

>
> I read no spaghetti code in my benchmark example. You are
> practicing poorly veiled deceit as a premise for trolling.
>
> My example exemplifies lexical scoping has an efficiency penalty.
>


Actually, it doesn't. I ran it a few times for a laugh and each time it
alternated which was faster. Moreover, if you did an accurate test and also
placed your globals inside a block (since a block does not enforce scoping,
using strictures and my variables does), your global example runs
considerably slower. This proves nothing though, except that you aren't very
good at benchmarking.

Not bothering to scope your variables (and that means *limiting* the scope)
might be fine for short scripts, but is terrible programming practice in
general. Programming is rarely about squeezing every second out of your
code, and certainly not at the expense of churning out spaghetti. And if you
can't look at your simple example and see that the global example is well on
the path to spaghetti-dom, then in the words of someone here, that's your
problem, not mine.

Matt


 
Reply With Quote
 
Jasper
Guest
Posts: n/a
 
      07-07-2004
Purl Gurl <(E-Mail Removed)> wrote in message news:<(E-Mail Removed)>...
> Purl Gurl wrote:
>
> > Jasper wrote:

>
> (snipped)
>
> > > I am clearing the contents of the arrays with @array = (); for
> > > all arrays, and I print out the $#array to get the size while the
> > > program is running for all arrays...and the arrays are not growing..so
> > > I am stumped as to what could be causing the fact that memory needs
> > > continue to increase...I am using GMP::BigFloat and doing large
> > > precision subtractions..

>
> > Try lexical scoping for your arrays, within private blocks,
> > to release memory for reuse.

>
>
> However, what none will tell you is although lexical scoping
> releases memory for reuse, you will trade some efficiency for
> better memory management.
>
> Use of lexical scoping will decrease efficiency by percentages
> which are easy to measure.
>
> When memory is an issue, lexical scoping is useful, otherwise
> a programmer should question if lexical scoping is beneficial
> within a context of efficiency.
>
>
> Purl Gurl
> --
> #!perl
>
> print "Content-type: text/plain\n\n";
>
> use Benchmark;
>
> print "Run One:\n\n";
> &Time;
>
> print "\n\nRun Two:\n\n";
> &Time;
>
> print "\n\nRun Three:\n\n";
> &Time;
>
> sub Time
> {
> timethese (100000,
> {
> 'Purl_Gurl' =>
> '@Array1 = qw (Now is the time for all good men to come to the aid of their country);
> @Array2 = qw (Behind every successful man walks a woman pushing him along);
> $var1 = "@Array1";
> @Array1 = ();
> $var2 = "@Array2";
> @Array2 = ();',
>
> 'Purl_Gurl2' =>
> '{
> my @Array3 = qw (Now is the time for all good men to come to the aid of their country);
> my $var3 = "@Array3";
> @Array3= ();
> }
> {
> my @Array4 = qw (Behind every successful man walks a woman pushing him along);
> my $var4 = "@Array4";
> @Array4= ();
> }',
> } );
> }
>
>
>
> PRINTED RESULTS:
> ________________
>
> Run One:
>
> Benchmark: timing 100000 iterations of Purl_Gurl, Purl_Gurl2...
> Purl_Gurl: 6 wallclock secs ( 5.43 usr + 0.00 sys = 5.43 CPU) @ 18416.21/s
> Purl_Gurl2: 6 wallclock secs ( 5.66 usr + 0.00 sys = 5.66 CPU) @ 17667.84/s
>
>
> Run Two:
>
> Benchmark: timing 100000 iterations of Purl_Gurl, Purl_Gurl2...
> Purl_Gurl: 6 wallclock secs ( 5.49 usr + 0.00 sys = 5.49 CPU) @ 18214.94/s
> Purl_Gurl2: 6 wallclock secs ( 5.77 usr + 0.00 sys = 5.77 CPU) @ 17331.02/s
>
>
> Run Three:
>
> Benchmark: timing 100000 iterations of Purl_Gurl, Purl_Gurl2...
> Purl_Gurl: 5 wallclock secs ( 5.44 usr + 0.00 sys = 5.44 CPU) @ 18382.35/s
> Purl_Gurl2: 4 wallclock secs ( 5.76 usr + 0.00 sys = 5.76 CPU) @ 17361.11/s


Thanks...
Great, but I have a subroutine which populates my arrays and then
OTHER subroutines analyze the data which are in the arrays...I cant
think of a way to make the arrays local....the whole point of
subroutines is modularity..how do folks conquer this with their
programs ? I cant copy large sections of code redundantly everywhere
! I dont know C so I am stuck...Can I use some kind of Perl "malloc"
syntax ? any ideas ?

Here is an example of what I am doing:

I have an array with no special declaration in Sub X:
for ($xx=$lastElem;$xx>=($lastElem-($additive-1));$xx--)
{
push @bottomArray1, $dataSet_init[$xx];
}

And then using it in Sub Y:
$subtractVal = $bottomArray1[0]- $bottomArray1[1];

What can I do to get around this ?
 
Reply With Quote
 
Eugene Mikheyev
Guest
Posts: n/a
 
      07-07-2004
> Thanks...
> Great, but I have a subroutine which populates my arrays and then
> OTHER subroutines analyze the data which are in the arrays...I cant
> think of a way to make the arrays local....the whole point of
> subroutines is modularity..how do folks conquer this with their
> programs ? I cant copy large sections of code redundantly everywhere
> ! I dont know C so I am stuck...Can I use some kind of Perl "malloc"
> syntax ? any ideas ?
>
> Here is an example of what I am doing:
>
> I have an array with no special declaration in Sub X:
> for ($xx=$lastElem;$xx>=($lastElem-($additive-1));$xx--)
> {
> push @bottomArray1, $dataSet_init[$xx];
> }
>
> And then using it in Sub Y:
> $subtractVal = $bottomArray1[0]- $bottomArray1[1];
>
> What can I do to get around this ?


You should have shown a place where your @bottomArray is declared, and how
your subs are called.


 
Reply With Quote
 
Tassilo v. Parseval
Guest
Posts: n/a
 
      07-07-2004
Also sprach Purl Gurl:

> Purl Gurl wrote:
>
>> Jasper wrote:

>
> (snipped)
>
>> > I am clearing the contents of the arrays with @array = (); for
>> > all arrays, and I print out the $#array to get the size while the
>> > program is running for all arrays...and the arrays are not growing..so
>> > I am stumped as to what could be causing the fact that memory needs
>> > continue to increase...I am using GMP::BigFloat and doing large
>> > precision subtractions..

>
>> Try lexical scoping for your arrays, within private blocks,
>> to release memory for reuse.

>
>
> However, what none will tell you is although lexical scoping
> releases memory for reuse, you will trade some efficiency for
> better memory management.
>
> Use of lexical scoping will decrease efficiency by percentages
> which are easy to measure.
>
> When memory is an issue, lexical scoping is useful, otherwise
> a programmer should question if lexical scoping is beneficial
> within a context of efficiency.


[ benchmark snipped ]

You only measures the time it takes to create and populate those
variables. Not a very realistic scenario considering that programs
hardly ever only create them. They also want to access them.

When it comes to that, lexicals usually perform better than globals.
There are a few exceptions, notably $_ and @_, which get special
treatment by the perl core through the global symbol PL_defgv.

No such shortcut exists for ordinary globals, though. Perl first has to
get the corresponding glob from the symbol-table after which it must get
at the correct slot. Lexicals however just live on the pad as SVs.
All perl has to do is C-cast the variable to its correct type.

Tassilo
--
$_=q#",}])!JAPH!qq(tsuJ[{@"tnirp}3..0}_$;//::niam/s~=)]3[))_$-3(rellac(=_$({
pam{rekcahbus})(rekcah{lrePbus})(lreP{rehtonabus}) !JAPH!qq(rehtona{tsuJbus#;
$_=reverse,s+(?<=sub).+q#q!'"qq.\t$&."'!#+sexisexi ixesixeseg;y~\n~~dddd;eval
 
Reply With Quote
 
Joe Smith
Guest
Posts: n/a
 
      07-07-2004
Jasper wrote:

> Great, but I have a subroutine which populates my arrays and then
> OTHER subroutines analyze the data which are in the arrays...I cant
> think of a way to make the arrays local....the whole point of
> subroutines is modularity..how do folks conquer this with their
> programs ? I cant copy large sections of code redundantly everywhere
> ! I dont know C so I am stuck...Can I use some kind of Perl "malloc"
> syntax ? any ideas ?


{
my @bottomArray1;
X(\@bottomArray1);
Y(\@bottomArray1);
}
sub X {
$reference_to_array = shift;
push @{$reference_to_array},...
}
sub Y {
$array_ref = shift;
$subtractVal = $array_ref->[0] - $array_ref->[1];
...
}

If you use function prototypes, the \ can be eliminated from \@bottomArray1.
-Joe
 
Reply With Quote
 
Tassilo v. Parseval
Guest
Posts: n/a
 
      07-07-2004
Also sprach Purl Gurl:

> Tassilo v. Parseval wrote:
>
>> Purl Gurl wrote:
>> > Purl Gurl wrote:
>> >> Jasper wrote:

>
> (snipped)
>
>> >> Try lexical scoping for your arrays, within private blocks,
>> >> to release memory for reuse.

>
>> > However, what none will tell you is although lexical scoping
>> > releases memory for reuse, you will trade some efficiency for
>> > better memory management.

>
>> > When memory is an issue, lexical scoping is useful, otherwise
>> > a programmer should question if lexical scoping is beneficial
>> > within a context of efficiency.

>
>> [ benchmark snipped ]

>
>> You only measures the time it takes to create and populate those
>> variables. Not a very realistic scenario considering that programs
>> hardly ever only create them. They also want to access them.

>
> Actually, my benchmark does a bit more to closer reflect
> a simple script. Arrays are created, variables are set,
> arrays are set to null.


You don't show how long it takes to access single elements, for
instance.

> Difference in timing is contributable to creating private
> blocks and is contributed to reuse of memory allotments.
>
> Without private blocks, unique "malloc" are created for
> each array (and variables) during compilation.
>
> With private blocks, blocks are created and a "single" malloc
> is created flagged for reuse. My use of "single" is not to
> imply only one malloc is created in all cases.
>
> For my example extra processing is invoked by perl core to
> keep track of blocks and malloc reuse.


I don't understand anything of the above. What is a '"single" malloc'
which is 'flagged for reuse'? Furthermore, what are those 'private
blocks'?

>> When it comes to that, lexicals usually perform better than globals.
>> There are a few exceptions, notably $_ and @_, which get special
>> treatment by the perl core through the global symbol PL_defgv.

>
>> No such shortcut exists for ordinary globals, though. Perl first has to
>> get the corresponding glob from the symbol-table after which it must get
>> at the correct slot. Lexicals however just live on the pad as SVs.
>> All perl has to do is C-cast the variable to its correct type.

>
> Inherently, there are lots of exceptions, oddities and other
> externalities effecting efficiency. For my benchmark, difference
> in efficiency is so small it is basically insignificant.
>
> For large programs, efficiency difference is significant.


Yes, but not in the sense that you would expect:

#! /usr/bin/perl -w

use Benchmark qw(cmpthese);

cmpthese( -2, {
pkg => sub {
@array = (1 .. 10000);
for $e (@array) {
$e++;
}
},
lex => sub {
my @ary = (1 .. 10000);
for my $e (@ary) {
$e++;
}
},
});
__END__
Rate pkg lex
pkg 89.6/s -- -41%
lex 151/s 68% --

Is that what you meant with 'significant'?

> However, for large programs reuse of memory may prove to
> be more efficient than not using lexical scoping. I have
> not tested large programs with benchmark. Still, this
> noted loss in efficiency, logically would be expected
> to amplify with larger scripts.


'reuse of memory may prove to be more efficient than not using lexical
scoping'? Another one I don't understand.

Tassilo
--
$_=q#",}])!JAPH!qq(tsuJ[{@"tnirp}3..0}_$;//::niam/s~=)]3[))_$-3(rellac(=_$({
pam{rekcahbus})(rekcah{lrePbus})(lreP{rehtonabus}) !JAPH!qq(rehtona{tsuJbus#;
$_=reverse,s+(?<=sub).+q#q!'"qq.\t$&."'!#+sexisexi ixesixeseg;y~\n~~dddd;eval
 
Reply With Quote
 
Tassilo v. Parseval
Guest
Posts: n/a
 
      07-07-2004
Also sprach Purl Gurl:

> Tassilo v. Parseval wrote:
>
>> Purl Gurl wrote:
>> > Tassilo v. Parseval wrote:
>> >> Purl Gurl wrote:
>> >> > Purl Gurl wrote:
>> >> >> Jasper wrote:

>
> (snipped)
>
>> >> [ benchmark snipped ]

>
>> > Actually, my benchmark does a bit more to closer reflect
>> > a simple script. Arrays are created, variables are set,
>> > arrays are set to null.

>
>> You don't show how long it takes to access single elements, for
>> instance.

>
> That is unrelated. The point of my test is to exemplify
> identical code takes longer to run when lexical scoping
> is employed.
>
>
>> I don't understand anything of the above. What is a '"single" malloc'
>> which is 'flagged for reuse'? Furthermore, what are those 'private
>> blocks'?

>
> My hunch is you already know and are testing my knowledge.
>
> For lexical scoping, a single chunk of memory, reserved by
> perl core, can be reused many times. That is a single "malloc"
> serving memory needs multiple times.
>
> Simple example, you can push an array into that "malloc" then
> later destroy it via lexical scoping, followed by another array
> being pushed into that same "malloc" memory chunk.
>
> Another here used a very good expression of calling those
> memory chunks an "arena" which is more understandable;
> blocks of reserved memory for use.


Ah, that. Without having looked too closely, global (package) variables
use the same memory allocation strategy. That means that memory attached
to dynamic variables will be reused as well, provided you tell perl that
it is allowed to reuse it, which for instance happens through a
statement like this:

@array = (1 .. 10000);
...
@array = (); # the memory of the 10000 scalars reusable now

>> > Inherently, there are lots of exceptions, oddities and other
>> > externalities effecting efficiency. For my benchmark, difference
>> > in efficiency is so small it is basically insignificant.

>
>> > For large programs, efficiency difference is significant.

>
>> Yes, but not in the sense that you would expect:

>
>> Rate pkg lex
>> pkg 89.6/s -- -41%
>> lex 151/s 68% --
>>
>> Is that what you meant with 'significant'?

>
> "Inherently, there are lots of exceptions, oddities and other...."


As I was trying to tell you, lexical variables being quicker than
dynamic ones is not an exception, it's the rule.

>> 'reuse of memory may prove to be more efficient than not using lexical
>> scoping'? Another one I don't understand.

>
> In general, use of global variables lends to a faster
> running script. Lexical scoping, slower. Use of "my"
> declarations within private blocks increases activity
> by perl core, hence, slower script completion.


No, it doesn't. The Perl interpreter runs through fewer lines of C code
when using lexical values. The whole procedure of unwrapping the scalar
(or array or hash) from a glob isn't done. With a lexical, perl knows at
compile time which of the various C structures to use. When they are put
into the pad (which is a special AV that holds the lexicals for a given
scope), they are simply casted to an SV. This is exactly one line of C
code:

#define PAD_SETSV(po,sv) PL_curpad[po] = (sv)

And retrieving a, say, array from this pad is just a matter of doing

AV *ary = (AV*)PL_curpad[po];

Whereas you want to tell me that retrieving a dynamic variable is less
than subscripting a C array plus a typecast? After all, you said that
lexicals increase the activity in the Perl core.

> As pointed out previously, it is very possible better
> use of memory may actually lend to better efficiency.
> I am thinking in terms of high RAM usage creating a
> greater degree of paging / swap file usage, which
> is slower because of hard drive access.


It doesn't have anything to do with that. Lexicals are faster because
perl has to carry out less work. I don't understand how one could
possibly argue about that.

Of course, you are free to continue using globals and pretend that
they'll make your programs perform better. Even if they slow them down.

Tassilo
--
$_=q#",}])!JAPH!qq(tsuJ[{@"tnirp}3..0}_$;//::niam/s~=)]3[))_$-3(rellac(=_$({
pam{rekcahbus})(rekcah{lrePbus})(lreP{rehtonabus}) !JAPH!qq(rehtona{tsuJbus#;
$_=reverse,s+(?<=sub).+q#q!'"qq.\t$&."'!#+sexisexi ixesixeseg;y~\n~~dddd;eval
 
Reply With Quote
 
Jasper
Guest
Posts: n/a
 
      07-07-2004
Purl Gurl <(E-Mail Removed)> wrote in message news:<(E-Mail Removed)>...
> Jasper wrote:
>
> > Purl Gurl wrote:
> > > Purl Gurl wrote:
> > > > Jasper wrote:

>
> (snipped)
>
> > > > > I am stumped as to what could be causing the fact that memory needs
> > > > > continue to increase.

>
> > > > Try lexical scoping for your arrays, within private blocks,
> > > > to release memory for reuse.

>
>
> > Great, but I have a subroutine which populates my arrays and then
> > OTHER subroutines analyze the data which are in the arrays.

>
> Then you are in a position to use lexical scoping.
>
> Combine your subroutines into one. Use of if / else logic
> is more efficient than invoking subroutines.



Combine into 1 subroutine ? Are you joking ? The sub that populates
the arrays is called constantly in several areas, collapsing into 1
subroutine would mean I would need to paste the same in multiple
places within that routine...that makes no sense..please clarify why
this would work..this program is very iterative and more than 1 sub
makes sense, but doing this has compromised my memory..

Thanks
Jasper


>
> > I cant think of a way to make the arrays local.

>
> Not local, but rather lexically scoped.
>
> Perhaps you need to think more?
>
>
> > the whole point of subroutines is modularity.

>
> Modules are modularity. Subroutines are used simply
> to write one task which is invoked multiple times.
>
> > I cant copy large sections of code redundantly everywhere

>
> Copy and paste works well for this. However, you should
> debate if you want to repeat code or want to write a
> single piece of code which is more efficient.
>
>
> > for ($xx=$lastElem;$xx>=($lastElem-($additive-1));$xx--)
> > {
> > push @bottomArray1, $dataSet_init[$xx];
> > }

>
> > $subtractVal = $bottomArray1[0]- $bottomArray1[1];

>
>
> Not enough information for comment nor do I read
> any use of lexical scoping.
>
>
> Purl Gurl

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Program eating memory, but only on one machine? Per B. Sederberg Python 5 01-22-2007 08:14 PM
501 PIX "deny any any" "allow any any" Any Anybody? Networking Student Cisco 4 11-16-2006 10:40 PM
Fx 1.5.0.1 media player plug-in eating memory Faun Firefox 0 02-26-2006 04:32 PM
Nero eating up / hiding memory DVD Video 3 07-09-2005 01:54 AM
4DOM eating all my memory ewan Python 1 02-02-2004 12:27 AM



Advertisments