Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > Processing workload distribution

Reply
Thread Tools

Processing workload distribution

 
 
Ted
Guest
Posts: n/a
 
      02-25-2008
The machine is a single processor/quad core server running the latest
Windows Server.

The Perl we're using is the 64 bit build of 5.8.8 from Activestate.

I created a script that uses threads to launch a series of standalone
SQL scripts (I use system to invoke mysql to run my SQL scripts). I
had thought that system was spawning child processes, but it seems
that the script that launches them waits on each child before
launching the next. It is looking like the threads launches all
without wating for the next, which is closer to what I want.

However, regardless of what I have tried so far, all the hard work is
being done by one core, with the other three mostly sitting idle.

What can I do to distribute the workload evenly over all the cores in
the machine? The whole purpose of this machine was to serve as a
compute server for my colleagues and I, and in fact only two of us
have a need to access it, and this is so we can run heavy duty DB
analysis programs on it. The code works adequately, but it seems a
shame to have invested in a quad core processor based machine only to
have all the work done on only one core. I don't care if I have to
launch threads or child processes, as long as I can distribute the
work more evenly. Is what I am after possible?

Thanks

Ted

BTW: are there any 64 bit modules on CPAN? The repository for the 64
bit build that Activestate has provided seems to be empty.
 
Reply With Quote
 
 
 
 
Ben Morrow
Guest
Posts: n/a
 
      02-25-2008

Quoth Ted <(E-Mail Removed)>:
> The machine is a single processor/quad core server running the latest
> Windows Server.
>
> The Perl we're using is the 64 bit build of 5.8.8 from Activestate.
>
> I created a script that uses threads to launch a series of standalone
> SQL scripts (I use system to invoke mysql to run my SQL scripts).


In general it is easier to talk to the database directly using DBI than
to launch external mysql processes. In this case, however, you may find
it easier to make things run in parallel with an external command.

> I had thought that system was spawning child processes, but it seems
> that the script that launches them waits on each child before
> launching the next.


Yes, this is how system works. As a special case, on Windows only,
calling

my $pid = system 1, "mysql ...";

(with a literal 1 as the first argument) will spawn a new process and
*not* wait for it. You can use the returned pid as an argument to wait
or waitpid. Alternatively, for more control, you can use Win32:rocess
or IPC::Run.

> It is looking like the threads launches all without wating for the
> next, which is closer to what I want.


You would have to post your code for us to see what is happening here. I
think probably each thread is launching one child and waiting for it,
but since you have several threads you have several children.

> However, regardless of what I have tried so far, all the hard work is
> being done by one core, with the other three mostly sitting idle.


Are you *certain* you end up with more than one mysql process running at
a time? You should be able to check easily with Task Manager. If you
don't, then you need to fix that (or switch to using DBI to talk to the
database and Perl threads for parallelism). If you do, yet they are all
running on the same core, then something odd is happening: you will need
to show us a (minimal) script that runs two processes at the same time
that still end up on the same core.

> BTW: are there any 64 bit modules on CPAN? The repository for the 64
> bit build that Activestate has provided seems to be empty.


Modules on CPAN are (generally-speaking) architecture-neutral, given
that CPAN only holds C and Perl source. If ActiveState don't provide
64-bit ppms, then you can install pure-Perl modules directly from CPAN,
but XS modules will require a copy of the compiler used to build your
perl, probably a 64-bit version of MSVC. You may be able to get gcc
working (the 32-bit version of AS Perl has support for building modules
with gcc), but I don't know how well the 64-bit version is supported.

Ben

 
Reply With Quote
 
 
 
 
smallpond
Guest
Posts: n/a
 
      02-25-2008
On Feb 25, 11:23 am, Ted <(E-Mail Removed)> wrote:
> The machine is a single processor/quad core server running the latest
> Windows Server.
>
> The Perl we're using is the 64 bit build of 5.8.8 from Activestate.
>
> I created a script that uses threads to launch a series of standalone
> SQL scripts (I use system to invoke mysql to run my SQL scripts). I
> had thought that system was spawning child processes, but it seems
> that the script that launches them waits on each child before
> launching the next. It is looking like the threads launches all
> without wating for the next, which is closer to what I want.
>
> However, regardless of what I have tried so far, all the hard work is
> being done by one core, with the other three mostly sitting idle.
>
> What can I do to distribute the workload evenly over all the cores in
> the machine? The whole purpose of this machine was to serve as a
> compute server for my colleagues and I, and in fact only two of us
> have a need to access it, and this is so we can run heavy duty DB
> analysis programs on it. The code works adequately, but it seems a
> shame to have invested in a quad core processor based machine only to
> have all the work done on only one core. I don't care if I have to
> launch threads or child processes, as long as I can distribute the
> work more evenly. Is what I am after possible?
>
> Thanks
>
> Ted
>
> BTW: are there any 64 bit modules on CPAN? The repository for the 64
> bit build that Activestate has provided seems to be empty.



See perlthrtut and check your perl build with: perl -V
Look for a line like:

usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define

I believe that the last part indicates whether perl tries to use
multiple
CPUs.
 
Reply With Quote
 
xhoster@gmail.com
Guest
Posts: n/a
 
      02-25-2008
Ted <(E-Mail Removed)> wrote:
> The machine is a single processor/quad core server running the latest
> Windows Server.
>
> The Perl we're using is the 64 bit build of 5.8.8 from Activestate.
>
> I created a script that uses threads to launch a series of standalone
> SQL scripts (I use system to invoke mysql to run my SQL scripts). I
> had thought that system was spawning child processes, but it seems
> that the script that launches them waits on each child before
> launching the next.


That is what Perl's "system" does. It spawns a child process, and then
waits for it.

> It is looking like the threads launches all
> without wating for the next, which is closer to what I want.
>
> However, regardless of what I have tried so far, all the hard work is
> being done by one core, with the other three mostly sitting idle.


Generally (not always, but generally) the hard work of an SQL query
is done by the server, not the client. If your MySQL database server fails
to use all the cores in doing it's job, that is not a Perl problem, that is
a MySQL problem.

If you don't think this describes your situation, then launch the four
standalone SQL scripts separately from Windows. If you launch them this
way, are all cores used? If not, then you don't have a Perl problem.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.
 
Reply With Quote
 
Ben Morrow
Guest
Posts: n/a
 
      02-25-2008

Quoth smallpond <(E-Mail Removed)>:
>
> See perlthrtut and check your perl build with: perl -V
> Look for a line like:
>
> usethreads=define use5005threads=undef useithreads=define
> usemultiplicity=define
>
> I believe that the last part indicates whether perl tries to use
> multiple CPUs.


Please don't make things up and post them without verifying.
Multiplicity indicates whether it is possible to have several perl
interpreters in the same process: this is a requirement for ithreads (so
any perl with useithreads=define will also have usemultiplicity=define)
but is also used without ithreads for e.g. mod_perl. It has *nothing* to
do with use of multiple CPUs: this is determined by your OS's thread
scheduling policy.

Ben

 
Reply With Quote
 
Ted
Guest
Posts: n/a
 
      02-25-2008
On Feb 25, 12:23*pm, smallpond <(E-Mail Removed)> wrote:
> On Feb 25, 11:23 am, Ted <(E-Mail Removed)> wrote:
>
>
>
>
>
> > The machine is a single processor/quad core server running the latest
> > Windows Server.

>
> > The Perl we're using is the 64 bit build of 5.8.8 from Activestate.

>
> > I created a script that uses threads to launch a series of standalone
> > SQL scripts (I use system to invoke mysql to run my SQL scripts). *I
> > had thought that system was spawning child processes, but it seems
> > that the script that launches them waits on each child before
> > launching the next. *It is looking like the threads launches all
> > without wating for the next, which is closer to what I want.

>
> > However, regardless of what I have tried so far, all the hard work is
> > being done by one core, with the other three mostly sitting idle.

>
> > What can I do to distribute the workload evenly over all the cores in
> > the machine? *The whole purpose of this machine was to serve as a
> > compute server for my colleagues and I, and in fact only two of us
> > have a need to access it, and this is so we can run heavy duty DB
> > analysis programs on it. *The code works adequately, but it seems a
> > shame to have invested in a quad core processor based machine only to
> > have all the work done on only one core. *I don't care if I have to
> > launch threads or child processes, as long as I can distribute the
> > work more evenly. *Is what I am after possible?

>
> > Thanks

>
> > Ted

>
> > BTW: are there any 64 bit modules on CPAN? *The repository for the 64
> > bit build that Activestate has provided seems to be empty.

>
> See perlthrtut and check your perl build with: perl -V
> Look for a line like:
>
> usethreads=define use5005threads=undef useithreads=define
> usemultiplicity=define
>
> I believe that the last part indicates whether perl tries to use
> multiple
> CPUs.- Hide quoted text -
>
> - Show quoted text -


Yes, both on my development machine (only a dual core) and on the quad
core server, I see this: use of both threads and multiplicity are
there. Near the top of the output from perl -V, I see:

usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define

And lower down I see,

Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT
PERL_IMPLICIT_SYS PERL_MALLOC_WRAP
PL_OP_SLAB_ALLOC USE_ITHREADS USE_LARGE_FILES
USE_PERLIO USE_SITECUSTOMIZE

When I launch four mysql commands separately, I see four instances of
it in task manager, and the work load is distributed over all four
cores. So the problem must be in my code (See my reply to myself).

Thanks

Ted
 
Reply With Quote
 
Ted
Guest
Posts: n/a
 
      02-25-2008
On Feb 25, 12:26*pm, (E-Mail Removed) wrote:
> Ted <(E-Mail Removed)> wrote:
> > The machine is a single processor/quad core server running the latest
> > Windows Server.

>
> > The Perl we're using is the 64 bit build of 5.8.8 from Activestate.

>
> > I created a script that uses threads to launch a series of standalone
> > SQL scripts (I use system to invoke mysql to run my SQL scripts). *I
> > had thought that system was spawning child processes, but it seems
> > that the script that launches them waits on each child before
> > launching the next.

>
> That is what Perl's "system" does. *It spawns a child process, and then
> waits for it.
>
> > It is looking like the threads launches all
> > without wating for the next, which is closer to what I want.

>
> > However, regardless of what I have tried so far, all the hard work is
> > being done by one core, with the other three mostly sitting idle.

>
> Generally (not always, but generally) the hard work of an SQL query
> is done by the server, not the client. *If your MySQL database server fails
> to use all the cores in doing it's job, that is not a Perl problem, that is
> a MySQL problem.
>
> If you don't think this describes your situation, then launch the four
> standalone SQL scripts separately from Windows. *If you launch them this
> way, are all cores used? *If not, then you don't have a Perl problem.
>
> Xho
>
> --
> --------------------http://NewsReader.Com/--------------------
> The costs of publication of this article were defrayed in part by the
> payment of page charges. This article must therefore be hereby marked
> advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
> this fact.


yes, so I have a perl problem (or probably more accurately my use of
perl is the problem).

Thanks

Ted
 
Reply With Quote
 
Ted
Guest
Posts: n/a
 
      02-25-2008
Here is my code:

use threads;

my $thr0 = threads->create(\&dbspawn,'e5pf16.sql','e5pf16.log');
$thr0->join;
my $thr1 = threads->create(\&dbspawn,'gpf45.sql','gpf45.log');
$thr1->join;
my $thr2 = threads->create(\&dbspawn,'gpf46.sql','gpf46.log');
$thr2->join;
my $thr3 = threads->create(\&dbspawn,'gpf47.sql','gpf47.log');
$thr3->join;
my $thr4 = threads->create(\&dbspawn,'gpf48.sql','gpf48.log');
$thr4->join;
my $thr5 = threads->create(\&dbspawn,'gpf49.sql','gpf49.log');
$thr5->join;
my $thr6 = threads->create(\&dbspawn,'gpf50.sql','gpf50.log');
$thr6->join;

sub dbspawn {
local ($script,$log) = @_;
system("mysql -t -u rejbyers --password=jesakos yohan < $script >
$log") == 0 or die "$script failed!";
}

Is the problem the way I use "system"?

When this script is running, I only ever see one instance of mysql
running in task manager.

Is there a Win64:rocess, or will Win32:rocess work on a 64 bit
version of windows using the 64 bit build of perl?

Thanks

Ted
 
Reply With Quote
 
Joost Diepenmaat
Guest
Posts: n/a
 
      02-25-2008
Ted <(E-Mail Removed)> writes:

> Here is my code:
>
> use threads;
>
> my $thr0 = threads->create(\&dbspawn,'e5pf16.sql','e5pf16.log');
> $thr0->join;


you should join() all threads /after/ creating all of them. a join()
blocks untill the thread is done. In other words, move the ->join()
calls to the end of script.

--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/
 
Reply With Quote
 
xhoster@gmail.com
Guest
Posts: n/a
 
      02-25-2008
Ted <(E-Mail Removed)> wrote:
> Here is my code:
>
> use threads;
>
> my $thr0 = threads->create(\&dbspawn,'e5pf16.sql','e5pf16.log');
> $thr0->join;


Join means you wait for $thr0 to finish. Obviously this is going
to lead to serialization. The simple answer is to move all the joins
from where they are and put them after all of the creates have been done.
The *right* answer is probably to put the thread objects into an array,
then after that loop over that array doing joins, rather than using a bunch
of different variables to hold the objects.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Crash/Reboot with WorkLoad =?Utf-8?B?QW5kcmU=?= Windows 64bit 1 11-30-2005 02:20 PM
Server Workload! Arpan ASP .Net 3 11-27-2004 06:38 PM



Advertisments