Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > Creating a 'load simulator' by calling Perl Programs - or Forking?

Reply
Thread Tools

Creating a 'load simulator' by calling Perl Programs - or Forking?

 
 
pgodfrin
Guest
Posts: n/a
 
      04-03-2008
Greetings,

I would like to create a 'driver' (simulation) program that calls
existing perl programs, waits for them to complete and provides some
feedback. Ideally I would like to have the driver echo to the screen
that the simulation is still running and perhaps provide some
information. That may be too ambitious, but that is the basic idea.

I have a handful of perl programs that do things - read a database
(kudos to the DBI modules) and output a report, from a few lines to a
few thousand.

Question #1: What is the proper way to execute the perl programs?

I am familiar with exec() and system() and have done this:

system("sallmr1 -n50000 >/dev/null");
system("srngmr1 -sr -es -n10000 >/dev/null");
system("srngmr2 -sr -es -n10000 >/dev/null");

Which executes these three programs sequentially - or by adding '&' at
the end I can execute them at the same time. Of course the 'calling'
program cannot wait for the background tasks because of the shell
return.

Which leads me down the path of fork(). If I were to code the
following:

for(1..2)
{
if($pid=fork)
{
# parent
} elsif (defined $pid)
{
# child
exec("sallmr1 -n50000 >/dev/null");
} # end child (elsif)
} # end for loop
do { $x=wait; print "$x died\n";} until $x==-1;

Then I would get a pair of runs, and wait until each child dies.

Question #2: Is there a way to echo to the screen that the wait() is
happening and provide some indication of progress?

To clarify this question, I had originally tried:
do {$x=wait; print ".";} until $x==-1;

Thinking I would get a period printed during the loop. (silly me -
wait only comes back when a child process dies...).

Of course if this is much to complicated a way to so a simulated load,
then so be it - just a plain old fork and wait works fine... And I
could always remove the /dev/null and actually see the called programs
output, which will tell me that it's still running... But I was
curious if there is a more elegant or at least interesting way of
doing this?

thanks,
phil
 
Reply With Quote
 
 
 
 
Joost Diepenmaat
Guest
Posts: n/a
 
      04-03-2008
pgodfrin <(E-Mail Removed)> writes:

> Greetings,
>
> I would like to create a 'driver' (simulation) program that calls
> existing perl programs, waits for them to complete and provides some
> feedback. Ideally I would like to have the driver echo to the screen
> that the simulation is still running and perhaps provide some
> information. That may be too ambitious, but that is the basic idea.


It may take a bit of getting used to, but POE::Wheel::Run can do this
very nicely, including monitoring of the fork'd procesesses' STDOUT and
STDERR, catching signals etc. And knowing POE will be useful in many
event-based processes.

See:
http://search.cpan.org/~rcaputo/POE-...E/Wheel/Run.pm

and the section "Process Management" in the POE cookbook:
http://poe.perl.org/?POE_Cookbook


--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/
 
Reply With Quote
 
 
 
 
pgodfrin
Guest
Posts: n/a
 
      04-03-2008
On Apr 3, 3:30 pm, Joost Diepenmaat <(E-Mail Removed)> wrote:
> pgodfrin <(E-Mail Removed)> writes:
> > Greetings,

>
> > I would like to create a 'driver' (simulation) program that calls
> > existing perl programs, waits for them to complete and provides some
> > feedback. Ideally I would like to have the driver echo to the screen
> > that the simulation is still running and perhaps provide some
> > information. That may be too ambitious, but that is the basic idea.

>
> It may take a bit of getting used to, but POE::Wheel::Run can do this
> very nicely, including monitoring of the fork'd procesesses' STDOUT and
> STDERR, catching signals etc. And knowing POE will be useful in many
> event-based processes.
>
> See:http://search.cpan.org/~rcaputo/POE-...E/Wheel/Run.pm
>
> and the section "Process Management" in the POE cookbook:http://poe.perl.org/?POE_Cookbook
>
> --
> Joost Diepenmaat | blog:http://joost.zeekat.nl/| work:http://zeekat.nl/


I was looking at IPC::Run as well. What do you think of it?
pg
 
Reply With Quote
 
Joost Diepenmaat
Guest
Posts: n/a
 
      04-03-2008
pgodfrin <(E-Mail Removed)> writes:

> I was looking at IPC::Run as well. What do you think of it?


Haven't used it all. And to be honest, from skimming the docs I can't
really see if it will even do what you want it to do if you have more
than one child process running, which I can guarantee you POE will.

But as I implied, POE has a bit of a learning curve, and it requires you
to (re)structure your code in a event handling way. YMMV, probably

--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/
 
Reply With Quote
 
pgodfrin
Guest
Posts: n/a
 
      04-03-2008
On Apr 3, 3:49 pm, Joost Diepenmaat <(E-Mail Removed)> wrote:
> pgodfrin <(E-Mail Removed)> writes:
> > I was looking at IPC::Run as well. What do you think of it?

>
> Haven't used it all. And to be honest, from skimming the docs I can't
> really see if it will even do what you want it to do if you have more
> than one child process running, which I can guarantee you POE will.
>
> But as I implied, POE has a bit of a learning curve, and it requires you
> to (re)structure your code in a event handling way. YMMV, probably
>
> --
> Joost Diepenmaat | blog:http://joost.zeekat.nl/| work:http://zeekat.nl/


Yeah - I agree. As I've mentioned in the past, I'm just a lowly DBA
(out of work btw) who's trying to not only learn Perl real good, but
use it simply to drive my databases so I can do DBA stuff - you know
like look at CPU util, I/O, funky SQL, etc.

So - I'm certain the POE will do the trick, on your say-so, but I'm
not sure if it is worth the learning curve at this juncture. But I
have bookmarked it and I will at least peruse it to see what it's all
about.

You never know...

smiles,
pg
 
Reply With Quote
 
xhoster@gmail.com
Guest
Posts: n/a
 
      04-03-2008
pgodfrin <(E-Mail Removed)> wrote:

....
> Question #1: What is the proper way to execute the perl programs?


You gave two examples, sequentially and in parallel. Which one is
appropriate depends on what kind of load you want to simulate, which in
turn depends on why are you are doing the simulation.


> Question #2: Is there a way to echo to the screen that the wait() is
> happening and provide some indication of progress?
>
> To clarify this question, I had originally tried:
> do {$x=wait; print ".";} until $x==-1;
>
> Thinking I would get a period printed during the loop. (silly me -
> wait only comes back when a child process dies...).


So you do get a period printed, once per child-exit. This is what I would
want to do, anyway, but I seem to be in a philosophical minority in this
regard (I hate progress indicators that merely indicate that the system
clock is ticking, and nothing about the actual progress of the actual
program.) I'm not sure what you originally thought it might do instead.
Periods to be printed as fast as Perl possible can print them?



> Of course if this is much to complicated a way to so a simulated load,
> then so be it - just a plain old fork and wait works fine... And I
> could always remove the /dev/null and actually see the called programs
> output, which will tell me that it's still running... But I was
> curious if there is a more elegant or at least interesting way of
> doing this?


If the waiting program is still running, and is still waiting, then the
program it is waiting on must still be running. What other possibilities
are there?

If you really want the "time has not come to a halt" thing, then
something like this:

perl -le '$|=1; fork or do {sleep 4*$_; exit} foreach (1..3); \
$SIG{ALRM}=sub{print "."; alarm 1}; alarm 1; \
do {$x=wait; print "$x";} until $x==-1; \
alarm 0'
..
..
..
20287
..
..
..
..
20288
..
..
..
..
20289
-1


Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.
 
Reply With Quote
 
pgodfrin
Guest
Posts: n/a
 
      04-04-2008
On Apr 3, 4:22 pm, (E-Mail Removed) wrote:
> pgodfrin <(E-Mail Removed)> wrote:
>
> ...
>
> > Question #1: What is the proper way to execute the perl programs?

>
> You gave two examples, sequentially and in parallel. Which one is
> appropriate depends on what kind of load you want to simulate, which in
> turn depends on why are you are doing the simulation.
>
> > Question #2: Is there a way to echo to the screen that the wait() is
> > happening and provide some indication of progress?

>
> > To clarify this question, I had originally tried:
> > do {$x=wait; print ".";} until $x==-1;

>
> > Thinking I would get a period printed during the loop. (silly me -
> > wait only comes back when a child process dies...).

>
> So you do get a period printed, once per child-exit. This is what I would
> want to do, anyway, but I seem to be in a philosophical minority in this
> regard (I hate progress indicators that merely indicate that the system
> clock is ticking, and nothing about the actual progress of the actual
> program.) I'm not sure what you originally thought it might do instead.
> Periods to be printed as fast as Perl possible can print them?
>
> > Of course if this is much to complicated a way to so a simulated load,
> > then so be it - just a plain old fork and wait works fine... And I
> > could always remove the /dev/null and actually see the called programs
> > output, which will tell me that it's still running... But I was
> > curious if there is a more elegant or at least interesting way of
> > doing this?

>
> If the waiting program is still running, and is still waiting, then the
> program it is waiting on must still be running. What other possibilities
> are there?
>
> If you really want the "time has not come to a halt" thing, then
> something like this:
>
> perl -le '$|=1; fork or do {sleep 4*$_; exit} foreach (1..3); \
> $SIG{ALRM}=sub{print "."; alarm 1}; alarm 1; \
> do {$x=wait; print "$x";} until $x==-1; \
> alarm 0'
> .
> .
> .
> 20287
> .
> .
> .
> .
> 20288
> .
> .
> .
> .
> 20289
> -1
>
> Xho
>
> --
> --------------------http://NewsReader.Com/--------------------
> The costs of publication of this article were defrayed in part by the
> payment of page charges. This article must therefore be hereby marked
> advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
> this fact.


Hi Xho,
As always - good points. I am looking to load up the system, so a
parallel simulation seems to be the best approach. And frankly, the
period printing was just a way to see that indeed something was
happening, but clearly and wholly unnecessary - although I did get a
chuckle over the time halting comment...

And anyway 'watch ps u' in anotehr session celarly shows me the
processes still running.

But - the question still remains - what's the best way to call other
perl programs? Exec, system or something else?

pg
 
Reply With Quote
 
xhoster@gmail.com
Guest
Posts: n/a
 
      04-04-2008
pgodfrin <(E-Mail Removed)> wrote:
>
> But - the question still remains - what's the best way to call other
> perl programs? Exec, system or something else?


I'd do fork and exec. If you want to do something in the child after
the other thing was done, then I'd do fork and system.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.
 
Reply With Quote
 
sheinrich@my-deja.com
Guest
Posts: n/a
 
      04-04-2008
On Apr 4, 6:12 am, pgodfrin <(E-Mail Removed)> wrote:
> On Apr 3, 4:22 pm, (E-Mail Removed) wrote:
>
>
>
> > pgodfrin <(E-Mail Removed)> wrote:

>
> > ...

>
> > > Question #1: What is the proper way to execute the perl programs?

>
> > You gave two examples, sequentially and in parallel. Which one is
> > appropriate depends on what kind of load you want to simulate, which in
> > turn depends on why are you are doing the simulation.

>
> > > Question #2: Is there a way to echo to the screen that the wait() is
> > > happening and provide some indication of progress?

>
> > > To clarify this question, I had originally tried:
> > > do {$x=wait; print ".";} until $x==-1;

>
> > > Thinking I would get a period printed during the loop. (silly me -
> > > wait only comes back when a child process dies...).

>
> > So you do get a period printed, once per child-exit. This is what I would
> > want to do, anyway, but I seem to be in a philosophical minority in this
> > regard (I hate progress indicators that merely indicate that the system
> > clock is ticking, and nothing about the actual progress of the actual
> > program.) I'm not sure what you originally thought it might do instead.
> > Periods to be printed as fast as Perl possible can print them?

>
> > > Of course if this is much to complicated a way to so a simulated load,
> > > then so be it - just a plain old fork and wait works fine... And I
> > > could always remove the /dev/null and actually see the called programs
> > > output, which will tell me that it's still running... But I was
> > > curious if there is a more elegant or at least interesting way of
> > > doing this?

>
> > If the waiting program is still running, and is still waiting, then the
> > program it is waiting on must still be running. What other possibilities
> > are there?

>
> > If you really want the "time has not come to a halt" thing, then
> > something like this:

>
> > perl -le '$|=1; fork or do {sleep 4*$_; exit} foreach (1..3); \
> > $SIG{ALRM}=sub{print "."; alarm 1}; alarm 1; \
> > do {$x=wait; print "$x";} until $x==-1; \
> > alarm 0'
> > .
> > .
> > .
> > 20287
> > .
> > .
> > .
> > .
> > 20288
> > .
> > .
> > .
> > .
> > 20289
> > -1

>
> > Xho

>
> > --
> > --------------------http://NewsReader.Com/--------------------
> > The costs of publication of this article were defrayed in part by the
> > payment of page charges. This article must therefore be hereby marked
> > advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
> > this fact.

>
> Hi Xho,
> As always - good points. I am looking to load up the system, so a
> parallel simulation seems to be the best approach. And frankly, the
> period printing was just a way to see that indeed something was
> happening, but clearly and wholly unnecessary - although I did get a
> chuckle over the time halting comment...
>
> And anyway 'watch ps u' in anotehr session celarly shows me the
> processes still running.
>
> But - the question still remains - what's the best way to call other
> perl programs? Exec, system or something else?
>
> pg


Hi pg,

a module that I've written couple of months ago was for that very
purpose.
Basically your script will have to instantiate an PreforkAgent, tell
it how many children to beget (degree of parallelization) and provide
it with a couple of callbacks as means of communication.
These callback routines may provide any task, execute it as you like
and perform evaluation of results and logging.
The module's initial purpose was it to generate as much load on a
remote system as the local server would allow. You can however, setup
a small number of children and even delay the task assignments without
interfering with the former tasks.
Children will live and be re-used until the job-queue is depleted.

You might download the module from http://www.atablis.com/temp/PreforkAgent.pm
and give it a try.

Cheers,
Steffen



It is rather easy to integrate
 
Reply With Quote
 
pgodfrin
Guest
Posts: n/a
 
      04-04-2008
On Apr 4, 1:44 am, (E-Mail Removed) wrote:
> On Apr 4, 6:12 am, pgodfrin <(E-Mail Removed)> wrote:
>
>
>
> > On Apr 3, 4:22 pm, (E-Mail Removed) wrote:

>
> > > pgodfrin <(E-Mail Removed)> wrote:

>
> > > ...

>
> > > > Question #1: What is the proper way to execute the perl programs?

>
> > > You gave two examples, sequentially and in parallel. Which one is
> > > appropriate depends on what kind of load you want to simulate, which in
> > > turn depends on why are you are doing the simulation.

>
> > > > Question #2: Is there a way to echo to the screen that the wait() is
> > > > happening and provide some indication of progress?

>
> > > > To clarify this question, I had originally tried:
> > > > do {$x=wait; print ".";} until $x==-1;

>
> > > > Thinking I would get a period printed during the loop. (silly me -
> > > > wait only comes back when a child process dies...).

>
> > > So you do get a period printed, once per child-exit. This is what I would
> > > want to do, anyway, but I seem to be in a philosophical minority in this
> > > regard (I hate progress indicators that merely indicate that the system
> > > clock is ticking, and nothing about the actual progress of the actual
> > > program.) I'm not sure what you originally thought it might do instead.
> > > Periods to be printed as fast as Perl possible can print them?

>
> > > > Of course if this is much to complicated a way to so a simulated load,
> > > > then so be it - just a plain old fork and wait works fine... And I
> > > > could always remove the /dev/null and actually see the called programs
> > > > output, which will tell me that it's still running... But I was
> > > > curious if there is a more elegant or at least interesting way of
> > > > doing this?

>
> > > If the waiting program is still running, and is still waiting, then the
> > > program it is waiting on must still be running. What other possibilities
> > > are there?

>
> > > If you really want the "time has not come to a halt" thing, then
> > > something like this:

>
> > > perl -le '$|=1; fork or do {sleep 4*$_; exit} foreach (1..3); \
> > > $SIG{ALRM}=sub{print "."; alarm 1}; alarm 1; \
> > > do {$x=wait; print "$x";} until $x==-1; \
> > > alarm 0'
> > > .
> > > .
> > > .
> > > 20287
> > > .
> > > .
> > > .
> > > .
> > > 20288
> > > .
> > > .
> > > .
> > > .
> > > 20289
> > > -1

>
> > > Xho

>
> > > --
> > > --------------------http://NewsReader.Com/--------------------
> > > The costs of publication of this article were defrayed in part by the
> > > payment of page charges. This article must therefore be hereby marked
> > > advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
> > > this fact.

>
> > Hi Xho,
> > As always - good points. I am looking to load up the system, so a
> > parallel simulation seems to be the best approach. And frankly, the
> > period printing was just a way to see that indeed something was
> > happening, but clearly and wholly unnecessary - although I did get a
> > chuckle over the time halting comment...

>
> > And anyway 'watch ps u' in anotehr session celarly shows me the
> > processes still running.

>
> > But - the question still remains - what's the best way to call other
> > perl programs? Exec, system or something else?

>
> > pg

>
> Hi pg,
>
> a module that I've written couple of months ago was for that very
> purpose.
> Basically your script will have to instantiate an PreforkAgent, tell
> it how many children to beget (degree of parallelization) and provide
> it with a couple of callbacks as means of communication.
> These callback routines may provide any task, execute it as you like
> and perform evaluation of results and logging.
> The module's initial purpose was it to generate as much load on a
> remote system as the local server would allow. You can however, setup
> a small number of children and even delay the task assignments without
> interfering with the former tasks.
> Children will live and be re-used until the job-queue is depleted.
>
> You might download the module fromhttp://www.atablis.com/temp/PreforkAgent.pm
> and give it a try.
>
> Cheers,
> Steffen
>
> It is rather easy to integrate


cool! I'll take a look...
phil
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Flash calling Php calling Perl Bill H Perl Misc 3 06-04-2009 03:32 PM
Book: Higher-Order Perl: Transforming Programs with Programs Casey Hawthorne Perl Misc 14 07-14-2005 06:43 PM
Creating a "Be Patient" window b4 calling a perl cgi script? Lisa Javascript 1 12-15-2004 10:48 AM
Creating a "Be Patient" window b4 calling a perl cgi script? Lisa Javascript 0 12-14-2004 10:54 PM
Calling programs from perl without temp files R Evans Perl 3 09-03-2004 10:18 AM



Advertisments