Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Perl Misc (http://www.velocityreviews.com/forums/f67-perl-misc.html)
-   -   Memory leak with threads (http://www.velocityreviews.com/forums/t913814-memory-leak-with-threads.html)

Jon Combe 09-16-2010 12:18 PM

Memory leak with threads
 
I have some Perl code that uses threads but found that after running
for a few days the memory usage had increased dramatically. I
simplified the code to the following but find when I run it with a
varying number of threads, the memory usage always increases. The
"countdown" at the end is just there to give me 30 seconds to grab the
memory utilisation before the process ends. I find that it uses approx
70MB for 50,000 threads, 140MB for 100,000 threads and 210MB for
150,000 threads.

I've tried on Perl 5.8.8, 10.2 and 12.2 all with similar results. Is
this a bug or have I just done something wrong? Any help appreciated!

#!/usr/bin/perl -w

use strict;
use threads;
use threads::shared;

for ( my $i = 0 ; $i < 150000 ; $i++ )
{
my $thread = threads->create('thread_function');
$thread -> join();
if ( $i % 1000 == 0 )
{
printf ( "%010i\n" , $i );
}
}

print "DONE\n";

for ( my $i = 0 ; $i < 30 ; $i++ )
{
print "Count down " . (30-$i) . "\n";
sleep(1);
}

sub thread_function
{
}

Thanks.
Jon

Dr.Ruud 09-16-2010 03:21 PM

Re: Memory leak with threads
 
On 2010-09-16 14:18, Jon Combe wrote:

> I have some Perl code that uses threads but found that after running
> for a few days the memory usage had increased dramatically. I
> simplified the code to the following but find when I run it with a
> varying number of threads, the memory usage always increases. The
> "countdown" at the end is just there to give me 30 seconds to grab the
> memory utilisation before the process ends. I find that it uses approx
> 70MB for 50,000 threads, 140MB for 100,000 threads and 210MB for
> 150,000 threads.
>
> I've tried on Perl 5.8.8, 10.2 and 12.2 all with similar results. Is
> this a bug or have I just done something wrong? Any help appreciated!


Joined threads still occupy memory, so that is probably all.

Play around with the below. Also run it with a parameter.


#!/usr/bin/perl -w
use strict;

use Devel::Size qw( total_size );

use threads;
use threads::shared;

my @thread;
for ( 0 .. 2999 ) {

push @thread, threads->create('thread_function');

$thread[ -1 ]->join();

@ARGV and $thread[ -1 ] = undef;

if ( @thread % 1000 == 0 ) {
printf "%s %s\n" , 0+ @thread, total_size( \@thread );
}
}

printf "%s %s\n" , 0+ @thread, total_size( \@thread );

sub thread_function {
# ...
}


Without the undeffing, it looks like this:

0 56
1000 88160
2000 176256
3000 268448

and with undeffing, like this:

0 56
1000 20144
2000 40240
3000 64432


--
Ruud

Dr.Ruud 09-16-2010 03:27 PM

Re: Memory leak with threads
 
On 2010-09-16 17:21, Dr.Ruud wrote:

> Without the undeffing, it looks like this:
>
> 0 56
> 1000 88160
> 2000 176256
> 3000 268448
>
> and with undeffing, like this:
>
> 0 56
> 1000 20144
> 2000 40240
> 3000 64432


That was the output of a slightly newer version:

#!/usr/bin/perl -w
use strict;

use Devel::Size qw( total_size );

use threads;
use threads::shared;

use constant NUM => 3000;

my @thread;
printf "%8s %8s\n" , 0+ @thread, total_size( \@thread );

for ( 1 .. NUM ) {

push @thread, threads->create('thread_function');
$thread[ -1 ] -> join();

@ARGV and $thread[ -1 ] = undef;

if ( ( NUM - @thread ) % 1000 == 0 ) {
printf "%8s %8s\n" , 0+ @thread, total_size( \@thread );
}
}

sub thread_function {
# ...
}

__END__


--
Ruud

Jon Combe 09-16-2010 03:45 PM

Re: Memory leak with threads
 
> That was the output of a slightly newer version:

Thanks for looking into this. I'm a little puzzled why the need to
undef the thread. Since my original code stored each thread in my
$thread in the loop $thread should go out of scope after each
iteration of the loop and should get automatically cleaned up. Is that
not the case?

Looking at your results (and I get similar) it appears threads still
use memory after they've ended. Is there no way to free up all the
memory a thread occupied once it has ended?

Thanks
Jon

Steve C 09-16-2010 05:42 PM

Re: Memory leak with threads
 
On 09/16/2010 08:18 AM, Jon Combe wrote:
> I have some Perl code that uses threads but found that after running
> for a few days the memory usage had increased dramatically. I
> simplified the code to the following but find when I run it with a
> varying number of threads, the memory usage always increases. The
> "countdown" at the end is just there to give me 30 seconds to grab the
> memory utilisation before the process ends. I find that it uses approx
> 70MB for 50,000 threads, 140MB for 100,000 threads and 210MB for
> 150,000 threads.
>
> I've tried on Perl 5.8.8, 10.2 and 12.2 all with similar results. Is
> this a bug or have I just done something wrong? Any help appreciated!
>


Is there any way to change your application to have a thread pool
instead of creating and destroying threads?

sln@netherlands.com 09-16-2010 10:19 PM

Re: Memory leak with threads
 
On Thu, 16 Sep 2010 05:18:58 -0700 (PDT), Jon Combe <jcombe@gmail.com> wrote:

>I have some Perl code that uses threads but found that after running
>for a few days the memory usage had increased dramatically. I
>simplified the code to the following but find when I run it with a
>varying number of threads, the memory usage always increases. The
>"countdown" at the end is just there to give me 30 seconds to grab the
>memory utilisation before the process ends. I find that it uses approx
>70MB for 50,000 threads, 140MB for 100,000 threads and 210MB for
>150,000 threads.
>
>I've tried on Perl 5.8.8, 10.2 and 12.2 all with similar results. Is
>this a bug or have I just done something wrong? Any help appreciated!
>
>#!/usr/bin/perl -w
>
>use strict;
>use threads;
>use threads::shared;
>
>for ( my $i = 0 ; $i < 150000 ; $i++ )
>{
> my $thread = threads->create('thread_function');
> $thread -> join();
> if ( $i % 1000 == 0 )
> {
> printf ( "%010i\n" , $i );
> }
>}
>
>print "DONE\n";
>
>for ( my $i = 0 ; $i < 30 ; $i++ )
>{
> print "Count down " . (30-$i) . "\n";
> sleep(1);
>}
>
>sub thread_function
>{
>}
>


I ran this code and did not find the problem you
have. I'm using Activestate 5.10.0 for win32 on an
XP platform.

From task manager, the mem usage fluxuates around 3.2 MB
(sort of a minimum for the perl interpreter). The # of
threads is never more than 1 - 2 so the thread is being
destroyed fine.

There is no reason "my $thread" should not be reused and its
object dereferenced.
What seems to be the problem is the object it references
is not being freed.

Some things you could try, but probably won't work:

- threads->create(\&thread_function)->join();

- Create just a few threads, then see what these say:
threads->list()
threads->list(threads::all)
threads->list(threads::running)
threads->list(threads::joinable)

Also, there are caveats about various things, stacksize,
OS, etc..

-sln

Xho Jingleheimerschmidt 09-17-2010 02:12 AM

Re: Memory leak with threads
 
Dr.Ruud wrote:
> On 2010-09-16 14:18, Jon Combe wrote:
>
>> I have some Perl code that uses threads but found that after running
>> for a few days the memory usage had increased dramatically. I
>> simplified the code to the following but find when I run it with a
>> varying number of threads, the memory usage always increases. The
>> "countdown" at the end is just there to give me 30 seconds to grab the
>> memory utilisation before the process ends. I find that it uses approx
>> 70MB for 50,000 threads, 140MB for 100,000 threads and 210MB for
>> 150,000 threads.
>>
>> I've tried on Perl 5.8.8, 10.2 and 12.2 all with similar results. Is
>> this a bug or have I just done something wrong? Any help appreciated!

>
> Joined threads still occupy memory, so that is probably all.
>
> Play around with the below. Also run it with a parameter.
>
>
> #!/usr/bin/perl -w
> use strict;
>
> use Devel::Size qw( total_size );
>
> use threads;
> use threads::shared;
>
> my @thread;
> for ( 0 .. 2999 ) {
>
> push @thread, threads->create('thread_function');


Why introduce an array when it wasn't there in the original code and it
doesn't do anything meaningful?


>
> $thread[ -1 ]->join();
>
> @ARGV and $thread[ -1 ] = undef;
>
> if ( @thread % 1000 == 0 ) {
> printf "%s %s\n" , 0+ @thread, total_size( \@thread );
> }
> }
>
> printf "%s %s\n" , 0+ @thread, total_size( \@thread );
>
> sub thread_function {
> # ...
> }
>
>
> Without the undeffing, it looks like this:
>
> 0 56
> 1000 88160
> 2000 176256
> 3000 268448
>
> and with undeffing, like this:
>
> 0 56
> 1000 20144
> 2000 40240
> 3000 64432
>



This is just telling us that arrays take up space up to their high water
mark, more when dense and less when sparse, but still space. It has
nothing to do with threads. The leak (which does seem to exist) is not
going to be detected using Devel::Size on a variable that has nothing to
do with the leak.

You need to measure the process space from the OS, not from Perl, which
after all likely cannot be trusted if it is leaking memory.

On Linux:

for ( my $i = 0 ; $i < 150000 ; $i++ )
{
my $thread = threads->create('thread_function');
$thread -> join();
if ( $i % 1000 == 0 )
{
printf ( "%010i\t%3\$s" , $i, `ps -p $$ -o rss` );
}
}


Xho

Jon Combe 09-17-2010 08:37 AM

Re: Memory leak with threads
 
> I ran this code and did not find the problem you
> have. I'm using Activestate 5.10.0 for win32 on an
> XP platform.
>
> From task manager, the mem usage fluxuates around 3.2 MB
> (sort of a minimum for the perl interpreter). The # of
> threads is never more than 1 - 2 so the thread is being
> destroyed fine.
>


Apologies I should have mentioned what OS I was using in my post. I'm
using Linux (Cent OS 5.4) and I've also tried Ubuntu Linux (10.2) with
the same result. However your results from Windows are interesting and
prompted me to try the same code on Windows. I have Starwberry Perl
rather than Active State and find that the memory stays constant at
around 3.8MB regardless of the number of threads. So it appears to be
an issue with Perl on Linux rather than Perl in general.

> There is no reason "my $thread" should not be reused and its
> object dereferenced.
> What seems to be the problem is the object it references
> is not being freed.
>
> Some things you could try, but probably won't work:
>
> - threads->create(\&thread_function)->join();
>
> - Create just a few threads, then see what these say:
> * threads->list()
> * threads->list(threads::all)
> * threads->list(threads::running)
> * threads->list(threads::joinable)
>
> Also, there are caveats about various things, stacksize,
> OS, etc..
>



I tried running the following:-

(Note for some reason I had to use threads::list not threads->list
even though the latter, which you suggested, is what it says to use in
the documentation)

#!/usr/bin/perl -w

use threads;
use threads::shared;

my @threads;
for ( my $j = 0 ; $j < 10 ; $j++ )
{
my $thread = threads->create(\&thread_function);
push ( @threads , $thread );
}
my @all_threads = threads::list(threads::all);
my @running_threads = threads::list(threads::running);
my @joinable_threads = threads::list(threads::joinable);
print "ALL:- : \n" . join ( "\n" , @all_threads ) . "\n";
print "RUNNING:- : \n" . join ( "\n" , @running_threads ) . "\n";
print "JOINABLE:- : \n" . join ( "\n" , @joinable_threads ) . "\n";

foreach ( @threads )
{
$_ -> join();
}

@all_threads = threads::list(threads::all);
@running_threads = threads::list(threads::running);
@joinable_threads = threads::list(threads::joinable);
print "ALL:- : \n" . join ( "\n" , @all_threads ) . "\n";
print "RUNNING:- : \n" . join ( "\n" , @running_threads ) . "\n";
print "JOINABLE:- : \n" . join ( "\n" , @joinable_threads ) . "\n";

sub thread_function
{

}

I get the following output:-

ALL:- :
threads::all=SCALAR(0x993fd0c)
threads::all=SCALAR(0x99cc024)
threads::all=SCALAR(0x99cc03c)
threads::all=SCALAR(0x99cc054)
threads::all=SCALAR(0x99cdc50)
threads::all=SCALAR(0x99cdc68)
threads::all=SCALAR(0x99cdc80)
threads::all=SCALAR(0x99cdc98)
threads::all=SCALAR(0x99cdcb0)
threads::all=SCALAR(0x99cdcc8)
RUNNING:- :
threads::running=SCALAR(0x99cc018)
threads::running=SCALAR(0x99cdd4c)
threads::running=SCALAR(0x99cdd64)
threads::running=SCALAR(0x99cdd7c)
threads::running=SCALAR(0x99cdd94)
threads::running=SCALAR(0x99cddac)
threads::running=SCALAR(0x99cddc4)
threads::running=SCALAR(0x99cdddc)
threads::running=SCALAR(0x99cddf4)
threads::running=SCALAR(0x99cde0c)
JOINABLE:- :
threads::joinable=SCALAR(0x99cdcbc)
threads::joinable=SCALAR(0x9da0e58)
threads::joinable=SCALAR(0x9da0e70)
threads::joinable=SCALAR(0x9da0e88)
threads::joinable=SCALAR(0x9da0ea0)
threads::joinable=SCALAR(0x9da0eb8)
threads::joinable=SCALAR(0x9da0ed0)
threads::joinable=SCALAR(0x9da0ee8)
threads::joinable=SCALAR(0x9da0f00)
threads::joinable=SCALAR(0x9da0f18)
ALL:- :

RUNNING:- :

JOINABLE:- :


So it does behave as I expect in that after doing the join no threads
are running.

Jon

Jon Combe 09-17-2010 08:40 AM

Re: Memory leak with threads
 
> You need to measure the process space from the OS, not from Perl, which
> after all likely cannot be trusted if it is leaking memory.
>
> On Linux:
>
> for ( my $i = 0 ; $i < 150000 ; $i++ )
> {
> * * * * *my $thread = threads->create('thread_function');
> * * * * *$thread -> join();
> * * * * *if ( $i % 1000 == 0 )
> * * * * *{
> * * * * * * * * *printf ( "%010i\t%3\$s" , $i, `ps -p $$ -o rss` );
> * * * * *}
>
> }


Here is a sample of the output I get from this - so it does confirm
for sure it's leaking memory.

0000000000 2872
0000001000 3672
0000002000 4384
0000003000 5092
0000004000 5804
0000005000 6516
0000006000 7228
0000007000 7936
0000008000 8648
0000009000 9356
0000010000 10072
0000011000 10784
0000012000 11496
0000013000 12204
0000014000 12920
0000015000 13628
0000016000 14340
0000017000 15052
0000018000 15764
0000019000 16428
0000020000 17184
0000021000 17900
0000022000 18616
0000023000 19328

Jon

Dr.Ruud 09-17-2010 10:25 AM

Re: Memory leak with threads
 
On 2010-09-17 10:40, Jon Combe wrote:

>> You need to measure the process space from the OS, not from Perl, which
>> after all likely cannot be trusted if it is leaking memory.
>>
>> On Linux:
>>
>> for ( my $i = 0 ; $i< 150000 ; $i++ )
>> {
>> my $thread = threads->create('thread_function');
>> $thread -> join();
>> if ( $i % 1000 == 0 )
>> {
>> printf ( "%010i\t%3\$s" , $i, `ps -p $$ -o rss` );
>> }
>>
>> }

>
> Here is a sample of the output I get from this - so it does confirm
> for sure it's leaking memory.
>
> 0000000000 2872
> 0000001000 3672
> 0000002000 4384
> 0000003000 5092
> 0000004000 5804
> 0000005000 6516
> 0000006000 7228
> 0000007000 7936
> 0000008000 8648
> 0000009000 9356
> 0000010000 10072
> 0000011000 10784
> 0000012000 11496
> 0000013000 12204
> 0000014000 12920
> 0000015000 13628
> 0000016000 14340
> 0000017000 15052
> 0000018000 15764
> 0000019000 16428
> 0000020000 17184
> 0000021000 17900
> 0000022000 18616
> 0000023000 19328


Also check perl -V |grep thread

With a perl 5.8.5. and archname=i386-linux-thread-multi, I get:

0000000000 2748
0000001000 2892
0000002000 2892
0000003000 2892
0000004000 2892
0000005000 2892

etc.

--
Ruud


All times are GMT. The time now is 10:16 AM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.