Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > Processing workload distribution

Reply
Thread Tools

Processing workload distribution

 
 
Joost Diepenmaat
Guest
Posts: n/a
 
      02-25-2008
Ted <(E-Mail Removed)> writes:

> Good to know. Thanks. I didn't consider fork because i was under the
> impression that it doesn't work right on Windows.


With regards to issues, fork() and threads should be more or less
equivalent on (native, non-cygwin) windows perls, since on windows fork
is implemented using threads. I vaguely recall that there may still be
some issues with fork() on windows, but they should be subtle as long as
you're not relying on /exact/ POSIX semantics for fork on windows.

In any case, threads (on any platform) and fork() on windows both have
their more-or-less subtle issues, so if it works for you, just use it,
but if you run into unexplained issues, they're definitely one of places
trouble may be looming.

--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/
 
Reply With Quote
 
 
 
 
Joost Diepenmaat
Guest
Posts: n/a
 
      02-25-2008
Ted <(E-Mail Removed)> writes:

>> There is the Perl function named "system", then there are calls that perl
>> makes into the kernel, referred to as system calls. *They are different
>> but confusingly named. *Is your concern about the first or the latter?
>>

> I assumed the former was just a special case of the latter. I was
> concerned about the former, though, because at present I am merely
> launched child processes using it. But the latter would always
> create a nagging doubt, for when I need to make more interesting
> threads.


System calls, on Unix, are a fairly small set of calls (a couple of
hundred, exact number depending on the OS) that can be made from the
program into the kernel. Those system calls basically define the minimum
functionality that programs may need to interact with the OS (and via
the os, with each other).

In addition, there are library functions for C language programs, a lot
of which are built on top of the system calls.

For example, fork(), wait() and exec() are always (well, probably
always) system calls on Unix, but system() is generally a library call
built on top of fork(), wait() and exec(), just like in general malloc()
is implemented as a library function built on the sbrk() syscall, which
is why perl can be built with different malloc() implementations on most
systems. Programmers not building compilers or other programs that need
to interact with the system on a very low level usually won't care which
is which.

Note that some of perl's built-in functions that interact directly with
the OS are sometimes called "syscalls" or "system calls" but may be
implemented in one of the standard C libraries or the perl binary itself
instead of directly in the OS kernel, and which is which may be
different on different OSs and perl builds.

--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/
 
Reply With Quote
 
 
 
 
Ben Morrow
Guest
Posts: n/a
 
      02-25-2008

Quoth smallpond <(E-Mail Removed)>:
> On Feb 25, 1:07 pm, Ben Morrow <(E-Mail Removed)> wrote:
> > Quoth smallpond <(E-Mail Removed)>:
> >
> > > See perlthrtut and check your perl build with: perl -V
> > > Look for a line like:

> >
> > > usethreads=define use5005threads=undef useithreads=define
> > > usemultiplicity=define

> >
> > > I believe that the last part indicates whether perl tries to use
> > > multiple CPUs.

> >
> > Please don't make things up and post them without verifying.
> > Multiplicity indicates whether it is possible to have several perl
> > interpreters in the same process: this is a requirement for ithreads (so
> > any perl with useithreads=define will also have usemultiplicity=define)
> > but is also used without ithreads for e.g. mod_perl. It has *nothing* to
> > do with use of multiple CPUs: this is determined by your OS's thread
> > scheduling policy.

>
> I didn't make it up, I looked at previous postings in this NG where
> someone has ithreads=define and not usemultiplicity.


You made up the bit about it affecting CPU affinity.

> http://groups.google.com/group/comp....se_frm/thread/
> 8f27b699aff8a37d/18a233467aaa72d8?hl=en&lnk=gst&q=usemultiplicity#
> 18a233467aaa72d8


[URL broken. Has the 'net collectively forgotten about msg-ids and news:
urls?]

From that posting:
| useithreads=define usemultiplicity=

I don't really know why that is; probably because the user didn't
explicitly ask for multiplicity. However, the important bits come later:

| Characteristics of this binary (from libperl):
| Compile-time options: DEBUGGING MULTIPLICITY USE_ITHREADS USE_LARGE_FILES
| PERL_IMPLICIT_CONTEXT

That list has both USE_ITHREADS and MULTIPLICITY in, so that perl was
built with both, as you can see from

| archname=i386-linux-thread-multi

right at the top.

Ben

 
Reply With Quote
 
Ben Morrow
Guest
Posts: n/a
 
      02-25-2008

Quoth Ted <(E-Mail Removed)>:
> On Feb 25, 1:51*pm, Ben Morrow <(E-Mail Removed)> wrote:
> >

[threads->create(sub { system "" }) vs.
async { system "" } vs.
system 1, "" on Win32
]
>
> Are there performance implications for the various forms you suggest,


No. async {...} is just a shortcut for threads->new(sub {...}).

> apart from the obvious that the last just has the overhead of creating
> the needed processes while the others create the same processes within
> threads that carry their own overhead. But I doubt that this overhead
> could be significant when the scripts that are launched take hours to
> complete.


I doubt *anything* you do in Perl will be significant compared with that
.

> > > Is there a Win64:rocess, or will Win32:rocess work on a 64 bit
> > > version of windows using the 64 bit build of perl?

> >
> > I don't know. That is, there isn't a Win64:rocess, and I don't know
> > whether Win32:rocess works on Win64. That depends on whether the
> > underlying Win32 system calls still work the same way: I suspect they
> > do. There are no Win64 results from CPAN Testers (surprise), which is
> > usually the first place to look. In any case, since there aren't any
> > Win64 ppms, you'd need to compile libwin32 from source. Do you have an
> > appropriate compiler?

>
> Well, I have MS Visual Studio 2003, so I suppose I could build it here
> (with a 64 bit target) and deploy it at the office.


Hmmm... maybe, maybe not. You need to use *exactly* the same compiler;
in particular, using a different version of MSVC from the one perl was
built with causes the msvcrt.dll linking to go haywire. I've no idea
what compiler ActiveState use for Win64: for Win32 they use MSVC 6,
which is compatible with MinGW gcc, but somehow I doubt MSVC 6 has a
64-bit target.

> I have yet to get cygwin to work on a 64 bit machine, so the 64 bit
> machine does not have a native compiler installed.


Oh, no, a cygwin compiler won't do at all.

> But wait a minute. On the 64 bit machine, 'ppm list' includes
> libwin32 at the bottom of the list. And, I see in the documentation
> Activestate provided with this build that Win32:rocess is available,
> along with a long list of other Win32 modules. Mind you, I don't know
> if it works since I just found it mere minutes ago.


D'oh! Of course, libwin32 has always been a standard part of ActivePerl
(that's where it came from in the first place). If it's there I'd expect
ActiveState to have tested it.

Ben

 
Reply With Quote
 
Ted
Guest
Posts: n/a
 
      02-25-2008
On Feb 25, 5:54*pm, Ben Morrow <(E-Mail Removed)> wrote:
> Quoth Ted <(E-Mail Removed)>:> On Feb 25, 1:51*pm, Ben Morrow <(E-Mail Removed)> wrote:
> > > > Is there a Win64:rocess, or will Win32:rocess work on a 64 bit
> > > > version of windows using the 64 bit build of perl?

>
> > > I don't know. That is, there isn't a Win64:rocess, and I don't know
> > > whether Win32:rocess works on Win64. That depends on whether the
> > > underlying Win32 system calls still work the same way: I suspect they
> > > do. There are no Win64 results from CPAN Testers (surprise), which is
> > > usually the first place to look. In any case, since there aren't any
> > > Win64 ppms, you'd need to compile libwin32 from source. Do you have an
> > > appropriate compiler?

>
> > Well, I have MS Visual Studio 2003, so I suppose I could build it here
> > (with a 64 bit target) and deploy it at the office.

>
> Hmmm... maybe, maybe not. You need to use *exactly* the same compiler;
> in particular, using a different version of MSVC from the one perl was
> built with causes the msvcrt.dll linking to go haywire. I've no idea
> what compiler ActiveState use for Win64: for Win32 they use MSVC 6,
> which is compatible with MinGW gcc, but somehow I doubt MSVC 6 has a
> 64-bit target.
>

While I have MSVS6, you could not pay me to install something that
antiquated on my system. Being used to a reasonably standard
compliant C++ compiler, using some that nacient and broken would be
much too painful.

> > I have yet to get cygwin to work on a 64 bit machine, so the 64 bit
> > machine does not have a native compiler installed.

>
> Oh, no, a cygwin compiler won't do at all.
>

Why not? Isn't it just another build of gcc, with libraries required
for programs to work on Windows?

> > But wait a minute. *On the 64 bit machine, 'ppm list' includes
> > libwin32 at the bottom of the list. *And, I see in the documentation
> > Activestate provided with this build that Win32:rocess is available,
> > along with a long list of other Win32 modules. *Mind you, I don't know
> > if it works since I just found it mere minutes ago.

>
> D'oh! Of course, libwin32 has always been a standard part of ActivePerl
> (that's where it came from in the first place). If it's there I'd expect
> ActiveState to have tested it.
>

Good to know.

Thanks

Ted

 
Reply With Quote
 
Joost Diepenmaat
Guest
Posts: n/a
 
      02-25-2008
Ted <(E-Mail Removed)> writes:

>> Oh, no, a cygwin compiler won't do at all.
>>

> Why not? Isn't it just another build of gcc, with libraries required
> for programs to work on Windows?


Cygwin's compiler is fine, if you're building for cygwin perl. If you're
building modules for activeperl you probably do need a VC compiler that
matches the one used by active state.

Another perl version that ships with a (free) compiler suite is
strawberry perl. It's pretty new, but you might get it to work for your
stuff.

--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/
 
Reply With Quote
 
smallpond
Guest
Posts: n/a
 
      02-26-2008
On Feb 25, 5:44 pm, Ben Morrow <(E-Mail Removed)> wrote:
> Quoth smallpond <(E-Mail Removed)>:
>
>
>
> > On Feb 25, 1:07 pm, Ben Morrow <(E-Mail Removed)> wrote:
> > > Quoth smallpond <(E-Mail Removed)>:

>
> > > > See perlthrtut and check your perl build with: perl -V
> > > > Look for a line like:

>
> > > > usethreads=define use5005threads=undef useithreads=define
> > > > usemultiplicity=define

>
> > > > I believe that the last part indicates whether perl tries to use
> > > > multiple CPUs.

>
> > > Please don't make things up and post them without verifying.
> > > Multiplicity indicates whether it is possible to have several perl
> > > interpreters in the same process: this is a requirement for ithreads (so
> > > any perl with useithreads=define will also have usemultiplicity=define)
> > > but is also used without ithreads for e.g. mod_perl. It has *nothing* to
> > > do with use of multiple CPUs: this is determined by your OS's thread
> > > scheduling policy.

>
> > I didn't make it up, I looked at previous postings in this NG where
> > someone has ithreads=define and not usemultiplicity.

>
> You made up the bit about it affecting CPU affinity.
>
> >http://groups.google.com/group/comp....se_frm/thread/
> > 8f27b699aff8a37d/18a233467aaa72d8?hl=en&lnk=gst&q=usemultiplicity#
> > 18a233467aaa72d8

>
> [URL broken. Has the 'net collectively forgotten about msg-ids and news:
> urls?]
>
> From that posting:
> | useithreads=define usemultiplicity=
>
> I don't really know why that is; probably because the user didn't
> explicitly ask for multiplicity. However, the important bits come later:
>
> | Characteristics of this binary (from libperl):
> | Compile-time options: DEBUGGING MULTIPLICITY USE_ITHREADS USE_LARGE_FILES
> | PERL_IMPLICIT_CONTEXT
>
> That list has both USE_ITHREADS and MULTIPLICITY in, so that perl was
> built with both, as you can see from
>
> | archname=i386-linux-thread-multi
>
> right at the top.
>
> Ben



The post is about this exact problem and a poster suggests
checking usemultiplicity. I inferred from that that they
were related since I see nothing in the perl documentation.

I find this in README.cygwin:
"Multiplicity is required when embedding Perl in a C program and using
more than one interpreter instance. This works with the Cygwin port."

It says nothing about ithreads. Maybe you made that up?
 
Reply With Quote
 
Ben Morrow
Guest
Posts: n/a
 
      02-26-2008

Quoth smallpond <(E-Mail Removed)>:
>
> The post is about this exact problem and a poster suggests
> checking usemultiplicity. I inferred from that that they
> were related since I see nothing in the perl documentation.


Then both the suggestion and your inference were incorrect. This is
understandable, as the documentation is a rather unclear in this area.

> I find this in README.cygwin:
> "Multiplicity is required when embedding Perl in a C program and using
> more than one interpreter instance. This works with the Cygwin port."
>
> It says nothing about ithreads. Maybe you made that up?


No. From perl.h:

#ifdef USE_ITHREADS
# if !defined(MULTIPLICITY)
# define MULTIPLICITY
# endif
#endif

so you can't have USE_ITHREADS without MULTIPLICITY. More obviously,
since multiplicity is what lets you have several Perl interpreters in
one process, and ithreads create a new interpreter for each thread, you
obviously can't have one without the other. And neither has anything to
do with thread scheduling, which, as I said, is entirely under the
control of your OS's thread implementation.

Ben

 
Reply With Quote
 
Ben Morrow
Guest
Posts: n/a
 
      02-26-2008

Quoth Joost Diepenmaat <(E-Mail Removed)>:
> Ted <(E-Mail Removed)> writes:
>
> >> Oh, no, a cygwin compiler won't do at all.
> >>

> > Why not? Isn't it just another build of gcc, with libraries required
> > for programs to work on Windows?

>
> Cygwin's compiler is fine, if you're building for cygwin perl. If you're
> building modules for activeperl you probably do need a VC compiler that
> matches the one used by active state.


32-bit ActivePerl is built with MSVC6. 32-bit MinGW gcc is
binary-compatible with MSVC6, and 32-bit ActivePerl ships with support
for using gcc to build extensions. Cygwin gcc, while essentially the
same compiler, has different default options (for instance, it links
with Cygwin's libc.dll by default) so it would be possible to make it
work, but difficult; and of course you'd need the MinGW import libs and
so on.

> Another perl version that ships with a (free) compiler suite is
> strawberry perl. It's pretty new, but you might get it to work for your
> stuff.


I looked at that first . Adam isn't yet shipping a 64-bit build. I
don't even know if gcc will build Win64 binaries.

Ben

 
Reply With Quote
 
xhoster@gmail.com
Guest
Posts: n/a
 
      02-26-2008
smallpond <(E-Mail Removed)> wrote:
> On Feb 25, 5:44 pm, Ben Morrow <(E-Mail Removed)> wrote:
> > Quoth smallpond <(E-Mail Removed)>:

>
> The post is about this exact problem and a poster suggests
> checking usemultiplicity.


I was that poster, and what I tentatively suggested was that
multiplicity looked kind of odd and perhaps should be looked into
as we had run out of more concrete ideas. Sometimes we bounce
ideas off each other here, which are intended to be less than the Word
of God.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Crash/Reboot with WorkLoad =?Utf-8?B?QW5kcmU=?= Windows 64bit 1 11-30-2005 02:20 PM
Server Workload! Arpan ASP .Net 3 11-27-2004 06:38 PM



Advertisments