Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Perl Misc (http://www.velocityreviews.com/forums/f67-perl-misc.html)
-   -   Perl threads (http://www.velocityreviews.com/forums/t903997-perl-threads.html)

ivakras1@gmail.com 07-30-2007 08:55 AM

Perl threads
 
Hi all!
I have a perl threads question. Of cuz i have read perl threads
tutorial and think its clear to me. But some questions...
Q1: Is threads->create method only creates thread, but not runs theads
until im join it (or detach)?
The folowing code and its output makes me think so:
=====================
....
for (my $i=0;$i<=10;$i++)
{
my $trl=threads->create(\&my_funct,$my_args);
if (@tc=threads->list){print "\t".@tc." threads running";}
}
.....
if (@tc=threads->list)
{
print "\nClosing threads: ";
while (@tc-1)
{
@tc=threads->list;
print @tc." ";
threads->list->join;
}
}
else
{
exit(0);
}

....
sub my_funct
{
$id=int(rand(1000));
setlogsock('unix'); #im using syslog to write a log file in /
var/log/messages
openlog('PerlSniffer','','debug'); #open a log
syslog('info', "Thread $id start"); #WRITE THREAD START TIME
$w=int(rand(5));
sleep($w); #SLEEP $w SECONDS
syslog('info', "Thread $id stop after $w s sleep"); #WRITE
THREAD STOP TIME
closelog;
return(1);
}
.....
=======================
The output of that program:

0 threads running
1 threads running
2 threads running
3 threads running
4 threads running
5 threads running
6 threads running
7 threads running
8 threads running
9 threads running
10 threads running
Closing threads: 10 9 8 7 6 5 4 3 2 1 0

=======================
/var/log/messages for that:

Jul 30 12:20:43 test_comp PerlSniffer: Thread 914 start
Jul 30 12:20:44 test_comp PerlSniffer: Thread 560 start
Jul 30 12:20:45 test_comp PerlSniffer: Thread 914 stop after 2 s sleep
Jul 30 12:20:45 test_comp PerlSniffer: Thread 560 stop after 1 s sleep
Jul 30 12:20:46 test_comp PerlSniffer: Thread 451 start
Jul 30 12:20:46 test_comp PerlSniffer: Thread 451 stop after 0 s sleep
Jul 30 12:20:48 test_comp PerlSniffer: Thread 303 start
Jul 30 12:20:48 test_comp PerlSniffer: Thread 303 stop after 0 s sleep
Jul 30 12:20:49 test_comp PerlSniffer: Thread 834 start
Jul 30 12:20:50 test_comp PerlSniffer: Thread 311 start
Jul 30 12:20:51 test_comp PerlSniffer: Thread 311 stop after 1 s sleep
Jul 30 12:20:52 test_comp PerlSniffer: Thread 677 start
Jul 30 12:20:52 test_comp PerlSniffer: Thread 677 stop after 0 s sleep
Jul 30 12:20:52 test_comp PerlSniffer: Thread 834 stop after 3 s sleep
Jul 30 12:20:52 test_comp PerlSniffer: Thread 133 start
Jul 30 12:20:52 test_comp PerlSniffer: Thread 133 stop after 0 s sleep
Jul 30 12:20:54 test_comp PerlSniffer: Thread 718 start
Jul 30 12:20:55 test_comp PerlSniffer: Thread 718 stop after 0 s sleep
Jul 30 12:20:56 test_comp PerlSniffer: Thread 909 start
Jul 30 12:20:58 test_comp PerlSniffer: Thread 520 start
Jul 30 12:21:00 test_comp PerlSniffer: Thread 909 stop after 4 s sleep
Jul 30 12:21:02 test_comp PerlSniffer: Thread 520 stop after 4 s sleep
=======================
Coments: i create 11 threads first, and than join them. Each thread
waits for 0-5 (rand(5)) seconds, and than exits, writes a start time
and a stop time in /var/log/messages via syslogd. It looks like thread-
>create only creates the thread, not runs it. The output shoes it:

threads->list increments every threads->create, but some of created
threads would be done during this time, if i wrong!!! When i threads-
>list->join one by one the threads seems to start, and threads->list

starts decreasing! Can anybody answer me?

Q2: threads->list returns a list of unjoind or undetached threads. How
to return a count of all threads, joined and detached both?? Then the
main program exits while some child threads are still running
(detached), perl warns me the count of running parentless threads. How
can i get that count during main program still alive?!

Thanks a lot and forgive me my baaad english, im russian actually.
Praying for the answers! Thanks again!


zentara 07-30-2007 11:11 AM

Re: Perl threads
 
On Mon, 30 Jul 2007 08:55:55 -0000, ivakras1@gmail.com wrote:

>Hi all!
>I have a perl threads question. Of cuz i have read perl threads
>tutorial and think its clear to me. But some questions...
>Q1: Is threads->create method only creates thread, but not runs theads
>until im join it (or detach)?


The thread starts running as soon as you create it.

There are some simple rules.
1. A thread, whether joinable or detached, MUST return, or reach the end
of it's code block, for it to terminate. That means you may need some
logic with shared variables, to signal a thread to return, if you want
to avoid the "a thread exited while x running" warning. The warning
is harmless, unless the threads were doing something that needed a clean
finish. Joining by itself, WILL NOT terminate a thread, unless it has
returned first.

2. Use alot of shared variables to track and control your threads.

3. You have 2 ways to get return values from a thread. One is through
shared variables, and the other is through the return array.


#!/usr/bin/perl
use threads;

# join() does three things: it waits for a thread to exit,
# cleans up after it, and returns any data the thread may
# have produced.

$thr1 = threads->new(\&sub1);
$ReturnData1 = $thr1->join;
print "Thread1 returned @$ReturnData1\n";

$thr2 = threads->new(\&sub2);
$ReturnData2 = $thr2->join;
print "Thread2 returned @$ReturnData2\n";

$thr3 = threads->new(\&sub3);
$ReturnData3 = $thr3->join;
print "Thread3 returned @$ReturnData3\n";


sub sub1 {
print "In thread1.....\n";
sleep 10;
@values = ('1a','1b', '1c');
return \@values;
}

sub sub2 {
print "In thread2.....\n";
sleep 10;
@values = ('2a','2b', '2c');
return \@values;
}

sub sub3 {
print "In thread3.....\n";
sleep 10;
@values = ('3a','3b', '3c');
return \@values;
}
__END__

############################################
Some pseudo-code for terminating early:

#in main thread
my $die:shared = 0;
for( 1 .. 10 ){$thr = threads->new(\&thread_code);}
....
....
if( $foobar ==42 ){ clean_exit() }

sub clean_exit{
$die =1;
$_->join for threads->list();

}


#in thread
sub thread_code{
for (1..10){
if( $die == 1){return}
print "$_\n";
sleep 1;
}

print "I'm at the end of my code block, and ready to be joined\n";

}


zentara


--
I'm not really a human, but I play one on earth.
http://zentara.net/japh.html

ivakras1@gmail.com 07-30-2007 12:13 PM

Re: Perl threads
 
Oh i see now... I can controll thread over shared vars. Ok.
In your example:
print "I'm at the end of my code block, and ready to be joined\n";
Is that mean thread waits to be joined or detached anyway? It never
dies by itself?

I need to fire up threads and forget about them, and continuing main
program, so i use detach. How can i count how much threads are running
now?
Tnx!


xhoster@gmail.com 07-30-2007 05:08 PM

Re: Perl threads
 
ivakras1@gmail.com wrote:
> Hi all!
> I have a perl threads question. Of cuz i have read perl threads
> tutorial and think its clear to me. But some questions...
> Q1: Is threads->create method only creates thread, but not runs theads
> until im join it (or detach)?
> The folowing code and its output makes me think so:
> =====================
> ...



Using strict is probably even more important under threads than it
is without threads. You should use strict.



> for (my $i=0;$i<=10;$i++)
> {
> my $trl=threads->create(\&my_funct,$my_args);
> if (@tc=threads->list){print "\t".@tc." threads running";}
> }
> ....
> if (@tc=threads->list)
> {
> print "\nClosing threads: ";
> while (@tc-1)
> {
> @tc=threads->list;
> print @tc." ";
> threads->list->join;
> }
> }
> else
> {
> exit(0);
> }
>
> ...
> sub my_funct
> {
> $id=int(rand(1000));
> setlogsock('unix'); #im using syslog to write a log file in /
> var/log/messages


Instead of putting in a comment telling us you are using syslog, you
should show us the code that actually "use"s syslog. (By which
you presumably mean Sys::Syslog?) Is Sys::Syslog thread safe?


> Coments: i create 11 threads first, and than join them. Each thread
> waits for 0-5 (rand(5)) seconds, and than exits, writes a start time
> and a stop time in /var/log/messages via syslogd. It looks like thread-
> >create only creates the thread, not runs it. The output shoes it:

> threads->list increments every threads->create, but some of created
> threads would be done during this time,


The criteria for a thread to show up in list is not that it be "running"

From the doc:
threads->list();
This will return a list of all non joined, non
detached threads.

So threads that are "finished" but still haven't been joined or detached
will be in the list.


if i wrong!!! When i threads-
> >list->join one by one the threads seems to start, and threads->list

> starts decreasing! Can anybody answer me?


threads->list doesn't do what you think it does.


>
> Q2: threads->list returns a list of unjoind or undetached threads.


Er, OK, so you do know this. Then why the confusion earlier?

> How
> to return a count of all threads, joined and detached both?? Then the
> main program exits while some child threads are still running
> (detached), perl warns me the count of running parentless threads. How
> can i get that count during main program still alive?!


The easy way is not to detach the threads. Then, just before the parent
exits, run through threads->list and join all of them. Or you could use
Thread::Running

Xho

--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB

ivakras1@gmail.com 07-31-2007 04:44 AM

Re: Perl threads
 
On 30 , 21:08, xhos...@gmail.com wrote:
> Instead of putting in a comment telling us you are using syslog, you
> should show us the code that actually "use"s syslog. (By which
> you presumably mean Sys::Syslog?) Is Sys::Syslog thread safe?


use Sys::Syslog qw( :DEFAULT setlogsock); #thats how im using syslog

> The easy way is not to detach the threads. Then, just before the parent
> exits, run through threads->list and join all of them. Or you could use
> Thread::Running


I working on network sniffer and packet analizer. The main program
starts to capture network packets with "use Net::Pcap". Each 100
packets captured - main program dumps packets in a file and starts a
new thread with arguments, contains a file name. Than main thread
continue sniffing, and the thread starts to analize the file. The idea
is to make main program work with no pauses for waiting joined thread
or something else, it causes packet missing. I tried to fire up
threads with detach (threads creates fast, 10-20 per second). But when
the main program is done (by ctrl+c or else) - perl warns me the
number of threads still running. How can i get this number by myself
to wait for detached threads dies?


zentara 07-31-2007 11:44 AM

Re: Perl threads
 
On Tue, 31 Jul 2007 04:44:12 -0000, ivakras1@gmail.com wrote:

>is to make main program work with no pauses for waiting joined thread
>or something else, it causes packet missing.

I would suspect that the startup time for the thread, is where the
packets are being lost. Especially if the system load is heavy, the
thread takes time to set itself up. You may need to rethink your
approach to this.

> I tried to fire up
>threads with detach (threads creates fast, 10-20 per second). But when
>the main program is done (by ctrl+c or else) - perl warns me the
>number of threads still running. How can i get this number by myself
>to wait for detached threads dies?


When you detach a thread, the main thread forgets about it, so there is
no way to get them without keeping track of them yourself.

Make a shared array or scalar, to keep track of your threads.
When you create a thread, push it's identifier ( use a hash ) onto
the array.
At the end of the thread code block, just before it exits, have it
remove it's name from the array.

Then you can tell how many threads you have running, by the
scalar count of the array.

If you want to terminate early, remember, you will need a "$die" shared
variable, to tell the remaining threads to return immediately, and then
you can cleanly exit. (This means your thread code block needs to be
written to check for $die often.

One word of warning, about spawning alot of detached threads containing
object code from some module......... you may very well will gain memory
as the script runs, if the cleanup of refcounts is not good. You may get
lucky, but it is something to watch out for. It is very dis-heartening
to spend a bunch of time getting it all to work, only to find it sucks
up ram like a vacuum cleaner. :-)

Personally, if speed is important, to avoid missing packets, I would
create a bank of reusable joinable threads, that you can cycle thru.
You may also want to ask this on http://perlmonks.org, where a fellow
named BrowserUk may be able to show you how to use thread->queue.
I'm a "crude crafter" and like to hammer all my stuff together with
shared variables and event loops, but there are some more sophisticated
techniques out there.

Here is an example of a reusable thread-bank. I create 3 threads, and
when they finish, they get pushed back into an array for the next
available job. It also shows you how to track your threads with a shared
hash.


#!/usr/bin/perl
use warnings;
use strict;
use threads;
use threads::shared;

#works on WindowsME ActiveState 5.8.6.811

my $data = shift || 'date'; #sample code to pass to thread

my %shash;
#share(%shash); #will work only for first level keys
my %hash;
my %workers;
my $numworkers = 3;

foreach my $dthread(1..$numworkers){
share ($shash{$dthread}{'go'});
share ($shash{$dthread}{'progress'});
share ($shash{$dthread}{'timekey'}); #actual instance of the thread
share ($shash{$dthread}{'frame_open'}); #open or close the frame
share ($shash{$dthread}{'handle'});
share ($shash{$dthread}{'data'});
share ($shash{$dthread}{'pid'});
share ($shash{$dthread}{'die'});

$shash{$dthread}{'go'} = 0;
$shash{$dthread}{'progress'} = 0;
$shash{$dthread}{'timekey'} = 0;
$shash{$dthread}{'frame_open'} = 0;
$shash{$dthread}{'handle'} = 0;
$shash{$dthread}{'data'} = $data;
$shash{$dthread}{'pid'} = -1;
$shash{$dthread}{'die'} = 0;
$hash{$dthread}{'thread'} = threads->new(\&work,$dthread);
}


use Tk;
use Tk::Dialog;

my $mw = MainWindow->new(-background => 'gray50');

my $lframe = $mw->Frame( -background => 'gray50',-borderwidth=>10 )
->pack(-side =>'left' ,-fill=>'y');
my $rframe = $mw->Frame( -background => 'gray50',-borderwidth=>10 )
->pack(-side =>'right',-fill =>'both' );

my %actives = (); #hash to hold reusable numbered widgets used for
downloads
my @ready = (); #array to hold markers indicating activity is needed

#make 3 reusable downloader widget sets-------------------------
foreach(1..$numworkers){
push @ready, $_;
#frames to hold indicator
$actives{$_}{'frame'} = $rframe->Frame( -background => 'gray50' );

$actives{$_}{'stopbut'} = $actives{$_}{'frame'}->Button(
-text => "Stop Worker $_",
-background => 'lightyellow',
-command => sub { } )->pack( -side => 'left', -padx => 10 );

$actives{$_}{'label1'} = $actives{$_}{'frame'} ->Label(
-width => 3,
-background => 'black',
-foreground => 'lightgreen',
-textvariable => \$shash{$_}{'progress'},
)->pack( -side => 'left' );

$actives{$_}{'label2'} = $actives{$_}{'frame'} ->Label(
-width => 1,
-text => '%',
-background => 'black',
-foreground => 'lightgreen',
)->pack( -side => 'left' );


$actives{$_}{'label3'} = $actives{$_}{'frame'} ->Label(
-text => '',
-background => 'black',
-foreground => 'skyblue',
)->pack( -side => 'left',-padx =>10 );

}
#--------------------------------------------------

my $button = $lframe->Button(
-text => 'Get a worker',
-background => 'lightgreen',
-command => sub { &get_a_worker(time) }
)->pack( -side => 'top', -anchor => 'n', -fill=>'x', -pady =>
20 );

my $text = $rframe->Scrolled("Text",
-scrollbars => 'ose',
-background => 'black',
-foreground => 'lightskyblue',
)->pack(-side =>'top', -anchor =>'n');

my $repeat;
my $startbut;
my $repeaton = 0;
$startbut = $lframe->Button(
-text => 'Start Test Count',
-background => 'hotpink',
-command => sub {
my $count = 0;
$startbut->configure( -state => 'disabled' );
$repeat = $mw->repeat(
100,
sub {
$count++;
$text->insert( 'end', "$count\n" );
$text->see('end');
}
);
$repeaton = 1;
})->pack( -side => 'top', -fill=>'x', -pady => 20);

my $stoptbut = $lframe->Button(
-text => 'Stop Count',
-command => sub {
$repeat->cancel;
$repeaton = 0;
$startbut->configure( -state => 'normal' );
})->pack( -side => 'top',-anchor => 'n', -fill=>'x', -pady => 20 );

my $exitbut = $lframe->Button(
-text => 'Exit',
-command => sub {

foreach my $dthread(keys %hash){
$shash{$dthread}{'die'} = 1;
$hash{$dthread}{'thread'}->join
}

if ($repeaton) { $repeat->cancel }
#foreach ( keys %downloads ) {
# #$downloads{$_}{'repeater'}->cancel;
#}
# $mw->destroy;
exit;
})->pack( -side => 'top',-anchor => 'n', -fill=>'x', -pady => 20
);


#dialog to get file url---------------------
my $dialog = $mw->Dialog(
-background => 'lightyellow',
-title => 'Get File',
-buttons => [ "OK", "Cancel" ]
);

my $hostl = $dialog->add(
'Label',
-text => 'Enter File Url',
-background => 'lightyellow'
)->pack();

my $hostd = $dialog->add(
'Entry',
-width => 100,
-textvariable => '',
-background => 'white'
)->pack();

$dialog->bind( '<Any-Enter>' => sub { $hostd->Tk::focus } );

my $message = $mw->Dialog(
-background => 'lightyellow',
-title => 'ERROR',
-buttons => [ "OK" ]
);

my $messagel = $message->add(
'Label',
-text => ' ',
-background => 'hotpink'
)->pack();

$mw->repeat(10, sub{
if(scalar @ready == $numworkers){return}

foreach my $set(1..$numworkers){
$actives{$set}{'label1'}->
configure(-text =>\$shash{$set}{'progress'});

if(($shash{$set}{'go'} == 0) and
($shash{$set}{'frame_open'} == 1))
{
my $timekey = $shash{$set}{'timekey'};
$workers{ $timekey }{'frame'}->packForget;
$shash{$set}{'frame_open'} = 0;
push @ready, $workers{$timekey}{'setnum'};
if((scalar @ready) == 3)
{ }
$workers{$timekey} = ();
delete $workers{$timekey};
}
}
});

$mw->MainLoop;
################################################## #################

sub get_a_worker {

my $timekey = shift;

$hostd->configure( -textvariable => \$data);
if ( $dialog->Show() eq 'Cancel' ) { return }

#----------------------------------------------
#get an available frameset
my $setnum;
if($setnum = shift @ready){print "setnum->$setnum\n"}
else{ print "no setnum available\n"; return}

$workers{$timekey}{'setnum'} = $setnum;
$shash{$setnum}{'timekey'} = $timekey;

$workers{$timekey}{'frame'} = $actives{$setnum}{'frame'};
$workers{$timekey}{'frame'}->pack(-side =>'bottom', -fill => 'both' );

$workers{$timekey}{'stopbut'} = $actives{$setnum}{'stopbut'};
$workers{$timekey}{'stopbut'}->configure(
-command => sub {
$workers{$timekey}{'frame'}->packForget;
$shash{ $workers{$timekey}{'setnum'} }{'go'} = 0;
$shash{ $workers{$timekey}{'setnum'} }{'frame_open'} = 0;
push @ready, $workers{$timekey}{'setnum'};
if((scalar @ready) == $numworkers)
{ }
$workers{$timekey} = ();
delete $workers{$timekey};
});

$workers{$timekey}{'label1'} = $actives{$setnum}{'label1'};
$workers{$timekey}{'label1'}->configure(
-textvariable => \$shash{$setnum}{'progress'},
);

$workers{$timekey}{'label2'} = $actives{$setnum}{'label2'};


$workers{$timekey}{'label3'} = $actives{$setnum}{'label3'};
$workers{$timekey}{'label3'}->configure(-text => $timekey);



$shash{$setnum}{'go'} = 1;
$shash{$setnum}{'frame_open'} = 1;
#--------end of get_file sub--------------------------
}

################################################## ################
sub work{
my $dthread = shift;
$|++;
while(1){
if($shash{$dthread}{'die'} == 1){ return };

if ( $shash{$dthread}{'go'} == 1 ){

eval( system( $shash{$dthread}{'data'} ) );

foreach my $num (1..100){
$shash{$dthread}{'progress'} = $num;
print "\t" x $dthread,"$dthread->$num\n";
select(undef,undef,undef, .5);
if($shash{$dthread}{'go'} == 0){last}
if($shash{$dthread}{'die'} == 1){ return };
}

$shash{$dthread}{'go'} = 0; #turn off self before returning
}else
{ select(undef,undef,undef,.1) }
}
}
################################################## ###################

__END__



--
I'm not really a human, but I play one on earth.
http://zentara.net/japh.html

ivakras1@gmail.com 08-01-2007 10:30 AM

Re: Perl threads
 
OMG...
I would rethink all... I need a help on my idea in part of scenario..
perlmonks may help me.
Tnx a lot all for help here! If anybody can help with ideas-please
email me!



All times are GMT. The time now is 06:17 AM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.