Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > Perl Multithreading and use of Perl Modules

Reply
Thread Tools

Perl Multithreading and use of Perl Modules

 
 
Nate
Guest
Posts: n/a
 
      07-06-2005
Hello all,
I'm trying to create a multithreaded program in Perl 5.8 on Linux.
I am trying to package up my code into a .pm file, but I would still
like to be able to do some shared memory between running threads.

I seem to be hitting a road block where I may want a subroutine that's
in the module to have access to a shared memory variable.
Because the module starts up in a seperate memory space I can't get
access to the shared memory variables that I'm declaring in the main
program.

In some cases I'm trying to simply read a shared variable, but in other
cases I would like to update data in a shared hash.


Does anyone have any suggestions as to how I could best accomplish
this?

Thanks,
Nate

 
Reply With Quote
 
 
 
 
A. Sinan Unur
Guest
Posts: n/a
 
      07-06-2005
"Nate" <(E-Mail Removed)> wrote in
news:(E-Mail Removed) oups.com:

> I'm trying to create a multithreaded program in Perl 5.8 on Linux.
> I am trying to package up my code into a .pm file, but I would still
> like to be able to do some shared memory between running threads.
>
> I seem to be hitting a road block where I may want a subroutine that's
> in the module to have access to a shared memory variable.
> Because the module starts up in a seperate memory space I can't get
> access to the shared memory variables that I'm declaring in the main
> program.
>
> In some cases I'm trying to simply read a shared variable, but in
> other cases I would like to update data in a shared hash.


You have not given enough information. It would be immensely helpful if
you could give a minimal example which does not do what you want, and
explain exactly what you would like it to do instead, you will have a
better chance of getting useful responses.

As it stands, trying to help you involves setting up a lot of
scaffolding based on guesswork which I doubt many people will be doing.

Sinan

--
A. Sinan Unur <(E-Mail Removed)>
(reverse each component and remove .invalid for email address)

comp.lang.perl.misc guidelines on the WWW:
http://mail.augustmail.com/~tadmc/cl...uidelines.html
 
Reply With Quote
 
 
 
 
Nate
Guest
Posts: n/a
 
      07-07-2005
> "Nate" <(E-Mail Removed)> wrote in
> news:(E-Mail Removed) oups.com:
>
> > I'm trying to create a multithreaded program in Perl 5.8 on Linux.
> > I am trying to package up my code into a .pm file, but I would still
> > like to be able to do some shared memory between running threads.


>
> You have not given enough information. It would be immensely helpful if
> you could give a minimal example which does not do what you want, and
> explain exactly what you would like it to do instead, you will have a
> better chance of getting useful responses.
>
> As it stands, trying to help you involves setting up a lot of
> scaffolding based on guesswork which I doubt many people will be doing.
>
> Sinan


Sure, more info is always helpful...

The sample below consists of two files:
mt_test.pl which is the main perl file, and
testmulti.pm which is the perl module.

I would like to find the best way to allow the subroutines in
testmulti.pm to access some shared variables made possible through
multithreading. If someone out there believes this isbad programming
practice, then please suggest the best way to get the info I'm looking
for in the script below.

The idea in this example is to run a counter in multiple threads. As
the counters run, I would like to have a shared variable incremented as
each counter thread executes it's iteration of the counter.

The code below is broken, in that I'm not sure how to get the
subroutines in testmulti.pm to access the shared variable from
mt_test.pl.
The tracker() subroutine needs to be able to check the shared variables
in order to report status. The join method only allows data to be
returned when the thread exits. However, I would like the tracker
process to be able to collect live data from the counter threads while
they are running. In reality, these threads will be running for minutes
or hours, and I would like to have the tracker process report data as
close to realtime as possible.

I have gotten this to run when all the subroutines are in the
mt_test.pl, but I would like to run the subroutines in a module for
repeated use among other perl programs.

When it runs in a single perl script it looks like this:
# ./mt_test.pl
Counter1 Counter4 Counter5 TOTAL
1 1
2 1 3
3 1 4
4 2 6
5 2 1 8


However, in the "modularized" state I only get this:
# ./mt_test.pl

Counter1 Counter4 Counter5 TOTAL
All Threads have closed, shutting down tracker
EXIT Values: 1: 0 2: 0 3: 0 Trk:
Attempt to free unreferenced scalar: SV 0x8c272bc, Perl interpreter:
0x8b75cf0 during global destruction.
Attempt to free unreferenced scalar: SV 0x8ba4f00, Perl interpreter:
0x8b13ed0 during global destruction.
Attempt to free unreferenced scalar: SV 0x8b431ac, Perl interpreter:
0x8ab1b58 during global destruction.
Attempt to free unreferenced scalar: SV 0x8adffa8, Perl interpreter:
0x89b3050 during global destruction.



<mt_test.pl>
#!/usr/bin/perl

# Threading requires use of the "threads" module
use threads;
# In order to use shared variables, you need to include the
"threads::shared" module
use threads::shared;
use testmulti qw(tracker counter);

## COUNTER EXAMPLE ##
# In this example we're going to start 3 threads of incrementing
counters
# each counter runs at a different time interval.
# The purpose of this example is to show how to use shared memory for
# constantly running threads

# Establish some shared variables for the counters
my %stats : shared; #This hash is used to track the stats of each
counter.
my $total : shared; #This string variable holds the total of all
counters and is writable via variable locking

my $end_test=9; #Here we define how many test intervals to run

# Let's run 3 counters of different sleep intervals
my @tests : shared = (1,2,5);

# Start the threads
# Here we are creating threads that will call the counter() subroutine.

# the second argument is the list of variables to pass to the
subroutine
# Syntax: $handle = threads->create(subroutine,list of args);
my $counter1 = threads->create("counter", "$tests[0],$end_test");
my $counter4 = threads->create("counter", $tests[1]);
my $counter5 = threads->create("counter", $tests[2]);


# Run status tracker in seperate thread
# We will use this thread to collect the data from the shared variables
# and print the results.
# "tracker" is the name of the subroutine to call
# "1" is the number of seconds to sleep in between checks

my $tracker1 = threads->create("tracker", "1,@tests");

# The join() method allows you to collect return data from the thread
when it closes
my $return1 = $counter1->join();
my $return4 = $counter4->join();
my $return5 = $counter5->join();
my $returnt = $tracker1->join();
print "EXIT Values: 1: $return1 2: $return4 3: $return5 Trk:
$returnt\n";

</mt_test.pl>





<testmulti.pm>

# Multithread test package
package testmulti;
use strict;
use base 'Exporter';
use vars qw(@ISA @EXPORT);
@ISA = ('Exporter');
@EXPORT = qw(&tracker &counter);

#use threads::shared;

#my %stats : shared; #This hash is used to track the stats of each
counter.
#my $total : shared; #This string variable holds the total of all
counters. We use this to show locking


# Subroutine that tracks the results of the running counters
sub tracker {
my ($interval, @tests) = @_;

# Take the count of the number of running processes and increase by
1 since it's 0 based
my $num_tests = $#tests;
$num_tests++;

#print column headers
print "Counter1 Counter4 Counter5 TOTAL\n" ;
# Start the while loop for this subroutine
while () {
# Pull the list of running threads and increment counter of
running threads
my (@running_threads) = threads->list();
my $num_threads = $#running_threads;
$num_threads++;
# Detect a thread is missing and alert
print "A Thread has shut down\n" if $num_threads < $num_tests;

# For the purposes of a clean shutdown, we want the tracker to
return to the main thread when it's done.
# It becomes done, when all of the other threads have shut down.
if ($num_threads == 1){
print "All Threads have closed, shutting down tracker\n";
return;
}

# Print counter result data, if there's anything to print
print
"\t$stats{$tests[0]}\t$stats{$tests[1]}\t$stats{$tests[2]}\t$total\n"
if $stats{$tests[0]};
sleep $interval;

}

}


#Simple subroutine for counting something
sub counter{
my ($interval, $end_test) = @_;
# This is an "internal counter" that will be used to generate a return
value
# I only use it here to simulate having some value to return.
my $int_ctr=0;

# Run the counter until we reach the max number of counter tests to
run, as defined by a global variable
while($total < $end_test){
print "\t\t\t\t\tCounter $interval reports the total as $total\n";
sleep $interval;
#lock($stats{$interval});
$stats{$interval}++;
# Lock the $total variable for writing. This causes other threads
to block while waiting for the lock to be released
# THe lock is released when the code context is exited at the end
of the BLOCK
lock($total);
$total++;
$int_ctr++;
}
# Return a known value so that we know what we're getting in the
join() method
return $int_ctr;

}
1;

</testmulti.pm>

 
Reply With Quote
 
A. Sinan Unur
Guest
Posts: n/a
 
      07-07-2005
"Nate" <(E-Mail Removed)> wrote in news:1120743693.919162.212330
@g43g2000cwa.googlegroups.com:

>> "Nate" <(E-Mail Removed)> wrote in
>> news:(E-Mail Removed) oups.com:
>>
>> > I'm trying to create a multithreaded program in Perl 5.8 on Linux.


....

>> You have not given enough information.


....

> The sample below consists of two files:
> mt_test.pl which is the main perl file, and
> testmulti.pm which is the perl module.


Thanks for posting that. I have not yet run anything but just a cursory
look makes me wonder about something. See below.

> <mt_test.pl>
> #!/usr/bin/perl


use strict;
use warnings;

missing.

> # Threading requires use of the "threads" module
> use threads;


Long comments like this tend to wrap, and make it difficult to get your
code to run.

> my $tracker1 = threads->create("tracker", "1,@tests");


....

> sub tracker {
> my ($interval, @tests) = @_;
>

....
> my $num_tests = $#tests;
> $num_tests++;


This is weird. You pass tracker only two arguments: The string
"tracker" and the *string* "1,1, 2, 5". That is, in the sub tracker,
@tests contains only one element.

I do not know if this has anything to do with your problem, but it seems
to be a significant error in logic.

Please also see

perldoc -q always.

Sinan
--
A. Sinan Unur <(E-Mail Removed)>
(reverse each component and remove .invalid for email address)

comp.lang.perl.misc guidelines on the WWW:
http://mail.augustmail.com/~tadmc/cl...uidelines.html
 
Reply With Quote
 
A. Sinan Unur
Guest
Posts: n/a
 
      07-07-2005
"Nate" <(E-Mail Removed)> wrote in news:1120743693.919162.212330
@g43g2000cwa.googlegroups.com:

> # Subroutine that tracks the results of the running counters
> sub tracker {
> my ($interval, @tests) = @_;

....
> my $num_tests = $#tests;
> $num_tests++;


In addition to my comments elsethread, you seem to be unaware that

my $num_tests = @tests;

would give the number of elements in @tests. In addition, unless you are
changing the number of elements in @tests, you do not need the variable
$num_tests. You can just use @tests in scalar context.

Sinan
--
A. Sinan Unur <(E-Mail Removed)>
(reverse each component and remove .invalid for email address)

comp.lang.perl.misc guidelines on the WWW:
http://mail.augustmail.com/~tadmc/cl...uidelines.html
 
Reply With Quote
 
Nate
Guest
Posts: n/a
 
      07-07-2005


A. Sinan Unur wrote:
> Thanks for posting that. I have not yet run anything but just a cursory
> look makes me wonder about something. See below.
>
> > <mt_test.pl>
> > #!/usr/bin/perl

>
> use strict;
> use warnings;
>
> missing.


Thanks.

>
> > # Threading requires use of the "threads" module
> > use threads;

>
> Long comments like this tend to wrap, and make it difficult to get your
> code to run.
>
> > my $tracker1 = threads->create("tracker", "1,@tests");


I'll remember that.

> > sub tracker {
> > my ($interval, @tests) = @_;
> >

> ...
> > my $num_tests = $#tests;
> > $num_tests++;

>
> This is weird. You pass tracker only two arguments: The string
> "tracker" and the *string* "1,1, 2, 5". That is, in the sub tracker,
> @tests contains only one element.
>
> I do not know if this has anything to do with your problem, but it seems
> to be a significant error in logic.


That was an error on my part. In other versions of the code, I passed
the list elements one by one. Thank you for pointing that error out. It
was affecting things.


>
> Please also see
>
> perldoc -q always.
>
> Sinan


More thanks. In looking at that I was able to figure out a way to do
what I needed.
It's been a little while since I've done any serious Perl programming,
and the use of references slipped my mind. I was able to rework the
code using references and have since been able to get the code to do
what I want.

Also, thank you for the reminder regarding the use of the scalar
context of the array to get the array count. I rememberted that $@array
would yield the number of elements -1, so I had been incremented them
to get the value!

I have fixed the code and it's working now. I'm going to clean it up a
little and post it here in a clean post.

Thanks for your help. Sorry for the newbie mistakes, I really have been
working in Perl for a little while, not that it shows.

-Nate

 
Reply With Quote
 
A. Sinan Unur
Guest
Posts: n/a
 
      07-07-2005
"Nate" <(E-Mail Removed)> wrote in
news:(E-Mail Removed) oups.com:

> Thanks for your help. Sorry for the newbie mistakes, I really have
> been working in Perl for a little while, not that it shows.


You are welcome. Don't worry about the mistakes. I do not point those out
to put anyone down, but to make sure the simple errors are fixed before
moving on to more complicated issues.

Please do post the new version with these errors fixed (and somewhat fewer
comments, it is easier to read code than comments in most cases), so
everyone can take a look to see if there are any other issues remaining.
For example, I would have passed a reference to %stats to the threads,
restricted locks to a smaller scope than the sub etc, but would not have
done more work before the obvious errors were fixed.

Sinan

--
A. Sinan Unur <(E-Mail Removed)>
(reverse each component and remove .invalid for email address)

comp.lang.perl.misc guidelines on the WWW:
http://mail.augustmail.com/~tadmc/cl...uidelines.html
 
Reply With Quote
 
Nate
Guest
Posts: n/a
 
      07-07-2005
Please see code Below. Thanks to Sinan's comments I was able to get
this to work as desired.

>>restricted locks to a smaller scope than the sub etc,

Sinan, what did you mean by this comment?


<mt_test.pl>
#!/usr/bin/perl
use strict;
use warnings;

use threads;
use threads::shared;
use testmulti qw(tracker counter);

my %stats : shared;
my $total : shared = 0;
my $end_test=9;

# run 3 counters of different sleep intervals
my @tests = (1,4,5);
my @args1 = ("$tests[0]",$end_test,\%stats,\$total);
my @args4 = ("$tests[1]",$end_test,\%stats,\$total);
my @args5 = ("$tests[2]",$end_test,\%stats,\$total);


# Start the threads
my $counter1 = threads->create('counter', @args1);
my $counter4 = threads->create('counter', @args4);
my $counter5 = threads->create('counter', @args5);


# Run status tracker in seperate thread
my $tracker1 =
threads->create('tracker',(1,scalar(@tests),\%stats,\$tota l));

# collect return data from the thread when it closes
my $return1 = $counter1->join();
my $return4 = $counter4->join();
my $return5 = $counter5->join();
my $returnt = $tracker1->join();
print "EXIT Values: 1: $return1 2: $return4 3: $return5 Trk:
$returnt\n";
</mt_test.pl>
# Multithread test package
package testmulti;
use strict;
use base 'Exporter';
use vars qw(@ISA @EXPORT);
@ISA = ('Exporter');
@EXPORT = qw(&tracker &counter);

# track the results of the running counters
sub tracker {
my ($interval,$num_tests,$ref_stats,$ref_total) = @_;
#print "tracker $interval $ref_stats $ref_total\n";

#print column headers
print "Counter1 Counter4 Counter5 TOTAL\n" ;
# Start the while loop for this subroutine
while () {
# Get list of running threads and increment counter
my (@running_threads) = threads->list();
# Detect a thread is missing and alert
print "A Thread has shut down\n" if scalar(@running_threads) <
$num_tests;

#Tracker should return to main thread when done.
# We're done when all other threads shut down.
if (scalar(@running_threads) == 1){
print "All Threads have closed, shutting down tracker\n";
return 1;
}

# Print counter result data
#print 'Keys:' . keys(%$ref_stats) . "\n";
print
"\t$ref_stats->{'1'}\t$ref_stats->{'4'}\t$ref_stats->{'5'}\t$$ref_total\n";
sleep $interval;

}

}


#Simple subroutine for counting something
sub counter{
my ($interval, $end_test, $ref_stats, $ref_total) = @_;
#print "recieved: $interval, $end_test, $ref_stats, $ref_total\n";

# internal counter that will generate the return value
my $int_ctr=0;

# Run the counter until we reach the max number of counter tests
while($$ref_total < $end_test){
#print "\t\t\t\t\tCounter $interval reports the total as
${$ref_total}\n";
sleep $interval;
#lock($stats{$interval});
@$ref_stats{$interval}++;
# Lock the $total variable for writing.
lock(${$ref_total});
${$ref_total}++;
$int_ctr++;
}
# test return value to join method
return $int_ctr;

}
1;

<testmulti.pm>

</testmulti.pm>

 
Reply With Quote
 
Nate
Guest
Posts: n/a
 
      07-07-2005
Huh, I forgot to paste the testmulti.pm

<testmulti.pm>
# Multithread test package
package testmulti;
use strict;
use base 'Exporter';
use vars qw(@ISA @EXPORT);
@ISA = ('Exporter');
@EXPORT = qw(&tracker &counter);

# track the results of the running counters
sub tracker {
my ($interval,$num_tests,$ref_stats,$ref_total) = @_;
#print "tracker $interval $ref_stats $ref_total\n";

#print column headers
print "Counter1 Counter4 Counter5 TOTAL\n" ;
# Start the while loop for this subroutine
while () {
# Get list of running threads and increment counter
my (@running_threads) = threads->list();
# Detect a thread is missing and alert
print "A Thread has shut down\n" if scalar(@running_threads) <
$num_tests;

#Tracker should return to main thread when done.
# We're done when all other threads shut down.
if (scalar(@running_threads) == 1){
print "All Threads have closed, shutting down tracker\n";
return 1;
}

# Print counter result data
#print 'Keys:' . keys(%$ref_stats) . "\n";
print
"\t$ref_stats->{'1'}\t$ref_stats->{'4'}\t$ref_stats->{'5'}\t$$ref_total\n";
sleep $interval;

}

}


#Simple subroutine for counting something
sub counter{
my ($interval, $end_test, $ref_stats, $ref_total) = @_;
#print "recieved: $interval, $end_test, $ref_stats, $ref_total\n";

# internal counter that will generate the return value
my $int_ctr=0;

# Run the counter until we reach the max number of counter tests
while($$ref_total < $end_test){
#print "\t\t\t\t\tCounter $interval reports the total as
${$ref_total}\n";
sleep $interval;
#lock($stats{$interval});
@$ref_stats{$interval}++;
# Lock the $total variable for writing.
lock(${$ref_total});
${$ref_total}++;
$int_ctr++;
}
# test return value to join method
return $int_ctr;

}
1;

</testmulti.pm>

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: Big projects divided into modules and sub-modules ImpalerCore C Programming 0 03-10-2011 03:01 PM
perl multithreading performance dniq00@gmail.com Perl Misc 16 09-01-2008 04:17 PM
Importing modules from within other modules Tobiah Python 2 09-14-2003 09:18 PM
imputils - import problem modules from sys.modules Remy Cool Python 1 08-27-2003 02:25 PM



Advertisments