Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Python 2.6, multiprocessing module and BSD

Reply
Thread Tools

Python 2.6, multiprocessing module and BSD

 
 
YouCanCallMeAl@gmail.com
Guest
Posts: n/a
 
      10-21-2008
It seems that the multiprocessing module in 2.6 is broken for *BSD;
I've seen issue 3770 regarding this. I'm curious if there are more
details on this issue since the posts in 3770 were a bit unclear. For
example, one post claimed that the problem was that sem_open isn't
implemented in *BSD, but it is available on FreeBSD 7 (I checked). I'd
be willing to help get this working if someone could point me in the
right direction.
 
Reply With Quote
 
 
 
 
Philip Semanchuk
Guest
Posts: n/a
 
      10-22-2008

On Oct 21, 2008, at 6:45 PM, http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:

> It seems that the multiprocessing module in 2.6 is broken for *BSD;
> I've seen issue 3770 regarding this. I'm curious if there are more
> details on this issue since the posts in 3770 were a bit unclear. For
> example, one post claimed that the problem was that sem_open isn't
> implemented in *BSD, but it is available on FreeBSD 7 (I checked). I'd
> be willing to help get this working if someone could point me in the
> right direction.



Hi Al,
I don't know anything about the multiprocessing module, but I have
some recent experience with the semaphores on FreeBSD 6 & 7 as a
result of implementing my posix_ipc module.

Since you mentioned sem_open, I assume you're talking about POSIX
semaphores, rather than System V semaphores, yes? On FreeBSD 6 & 7
sem_open exists and works, but support for said semaphores are listed
as "very experimental". I experienced a problem that looked like a bug
to me, and a rather big one at that. There's more detail here (scroll
down to the part about FreeBSD 6/7) including a link to the bug report
I filed against the FreeBSD kernel:

http://semanchuk.com/philip/posix_ipc/#platforms


HTH
Philip
 
Reply With Quote
 
 
 
 
Jesse Noller
Guest
Posts: n/a
 
      10-22-2008
On Tue, Oct 21, 2008 at 6:45 PM, <(E-Mail Removed)> wrote:
> It seems that the multiprocessing module in 2.6 is broken for *BSD;
> I've seen issue 3770 regarding this. I'm curious if there are more
> details on this issue since the posts in 3770 were a bit unclear. For
> example, one post claimed that the problem was that sem_open isn't
> implemented in *BSD, but it is available on FreeBSD 7 (I checked). I'd
> be willing to help get this working if someone could point me in the
> right direction.
> --
> http://mail.python.org/mailman/listinfo/python-list
>


The BSD issue was raised late in the cycle for 2.6. The problem is
that FBSD's support is "very experimental" as Phillip points out - and
OpenBSD doesn't even have them.

Due to the lateness of the issue and a finite amount of time I have to
work on things, I chose to disable support for this on the various
*BSDs until I can cook up a stable patch or have one provided by
someone more familiar with the inner workings of Free-BSD. OpenBSD
support is a non-starter.

Ideally, I would like to get this fixed and put on the 2.6 maint
branch ASAP, but I haven't had a chance to circle back to it.

Also note Nick's comment in that bug: "Unfortunately, our OpenBSD and
FreeBSD buildbots are so unreliable that they don't get much attention
when they go red"

Stable reliable buildbots and a few more volunteers more familiar with
BSDs might be a great and welcome addition to python-dev.

As for getting this working - I would love a patch. You are going to
want to start with python-trunk and look in setup.py. You are going to
want to adjust the flags the package uses:

elif platform in ('freebsd5', 'freebsd6', 'freebsd7', 'freebsd8'):
# FreeBSD's P1003.1b semaphore support is very experimental
# and has many known problems. (as of June 200
macros = dict( # FreeBSD
HAVE_SEM_OPEN=0,
HAVE_SEM_TIMEDWAIT=0,
HAVE_FD_TRANSFER=1,
)
libraries = []

You will also need to look at: Lib/multiprocessing/synchronize.py to
disable the import error - Modules/_multiprocessing/multiprocessing.h
will need to be updated for the proper ifdefs for the bsd(s) as well.
Finally, the core of the semaphore usage is in
Modules/_multiprocessing/semaphore.c

I apologize we/I could not get this in for 2.6

-jesse
 
Reply With Quote
 
YouCanCallMeAl@gmail.com
Guest
Posts: n/a
 
      10-22-2008
On Oct 21, 8:08*pm, Philip Semanchuk <(E-Mail Removed)> wrote:
> On Oct 21, 2008, at 6:45 PM, (E-Mail Removed) wrote:
>
> > It seems that the multiprocessing module in 2.6 is broken for *BSD;
> > I've seen issue 3770 regarding this. I'm curious if there are more
> > details on this issue since the posts in 3770 were a bit unclear. For
> > example, one post claimed that the problem was that sem_open isn't
> > implemented in *BSD, but it is available on FreeBSD 7 (I checked). I'd
> > be willing to help get this working if someone could point me in the
> > right direction.

>
> Hi Al,
> I don't know anything about the multiprocessing module, but I have *
> some recent experience with the semaphores on FreeBSD 6 & 7 as a *
> result of implementing my posix_ipc module.
>
> Since you mentioned sem_open, I assume you're talking about POSIX *
> semaphores, rather than System V semaphores, yes? On FreeBSD 6 & 7 *
> sem_open exists and works, but support for said semaphores are listed *
> as "very experimental". I experienced a problem that looked like a bug *
> to me, and a rather big one at that. There's more detail here (scroll *
> down to the part about FreeBSD 6/7) including a link to the bug report *
> I filed against the FreeBSD kernel:
>
> http://semanchuk.com/philip/posix_ipc/#platforms
>
> HTH
> Philip


Thanks for the background on FreeBSD semaphores. I read through the
link; it should help.
 
Reply With Quote
 
YouCanCallMeAl@gmail.com
Guest
Posts: n/a
 
      10-22-2008
On Oct 22, 8:11*am, "Jesse Noller" <(E-Mail Removed)> wrote:
> On Tue, Oct 21, 2008 at 6:45 PM, *<(E-Mail Removed)> wrote:
> > It seems that the multiprocessing module in 2.6 is broken for *BSD;
> > I've seen issue 3770 regarding this. I'm curious if there are more
> > details on this issue since the posts in 3770 were a bit unclear. For
> > example, one post claimed that the problem was that sem_open isn't
> > implemented in *BSD, but it is available on FreeBSD 7 (I checked). I'd
> > be willing to help get this working if someone could point me in the
> > right direction.
> > --
> >http://mail.python.org/mailman/listinfo/python-list

>
> The BSD issue was raised late in the cycle for 2.6. The problem is
> that FBSD's support is "very experimental" as Phillip points out - and
> OpenBSD doesn't even have them.
>
> Due to the lateness of the issue and a finite amount of time I have to
> work on things, I chose to disable support for this on the various
> *BSDs until I can cook up a stable patch or have one provided by
> someone more familiar with the inner workings of Free-BSD. OpenBSD
> support is a non-starter.
>
> Ideally, I would like to get this fixed and put on the 2.6 maint
> branch ASAP, but I haven't had a chance to circle back to it.
>
> Also note Nick's comment in that bug: "Unfortunately, our OpenBSD and
> FreeBSD buildbots are so unreliable that they don't get much attention
> when they go red"
>
> Stable reliable buildbots and a few more volunteers more familiar with
> BSDs might be a great and welcome addition to python-dev.
>
> As for getting this working - I would love a patch. You are going to
> want to start with python-trunk and look in setup.py. You are going to
> want to adjust the flags the package uses:
>
> * * * * elif platform in ('freebsd5', 'freebsd6', 'freebsd7', 'freebsd8'):
> * * * * * * # FreeBSD's P1003.1b semaphore support is very experimental
> * * * * * * # and has many known problems. (as of June 200
> * * * * * * macros = dict( * * * * * * * * *# FreeBSD
> * * * * * * * * HAVE_SEM_OPEN=0,
> * * * * * * * * HAVE_SEM_TIMEDWAIT=0,
> * * * * * * * * HAVE_FD_TRANSFER=1,
> * * * * * * * * )
> * * * * * * libraries = []
>
> You will also need to look at: Lib/multiprocessing/synchronize.py to
> disable the import error - Modules/_multiprocessing/multiprocessing.h
> will need to be updated for the proper ifdefs for the bsd(s) as well.
> Finally, the core of the semaphore usage is in
> Modules/_multiprocessing/semaphore.c
>
> I apologize we/I could not get this in for 2.6
>
> -jesse


This is exactly the sort of response I was hoping for. Thanks for the
additional background on the problem. I'll take a look at the code and
see if I can figure out a patch. I'll also read up on the buildbot
issue ( I think I saw a link about that)...I might have a stable co-
located FreeBSD box that could be used.
-Alan
 
Reply With Quote
 
Jesse Noller
Guest
Posts: n/a
 
      10-22-2008
On Wed, Oct 22, 2008 at 10:31 AM, <(E-Mail Removed)> wrote:
> On Oct 22, 8:11 am, "Jesse Noller" <(E-Mail Removed)> wrote:
>> On Tue, Oct 21, 2008 at 6:45 PM, <(E-Mail Removed)> wrote:
>> > It seems that the multiprocessing module in 2.6 is broken for *BSD;
>> > I've seen issue 3770 regarding this. I'm curious if there are more
>> > details on this issue since the posts in 3770 were a bit unclear. For
>> > example, one post claimed that the problem was that sem_open isn't
>> > implemented in *BSD, but it is available on FreeBSD 7 (I checked). I'd
>> > be willing to help get this working if someone could point me in the
>> > right direction.
>> > --
>> >http://mail.python.org/mailman/listinfo/python-list

>>
>> The BSD issue was raised late in the cycle for 2.6. The problem is
>> that FBSD's support is "very experimental" as Phillip points out - and
>> OpenBSD doesn't even have them.
>>
>> Due to the lateness of the issue and a finite amount of time I have to
>> work on things, I chose to disable support for this on the various
>> *BSDs until I can cook up a stable patch or have one provided by
>> someone more familiar with the inner workings of Free-BSD. OpenBSD
>> support is a non-starter.
>>
>> Ideally, I would like to get this fixed and put on the 2.6 maint
>> branch ASAP, but I haven't had a chance to circle back to it.
>>
>> Also note Nick's comment in that bug: "Unfortunately, our OpenBSD and
>> FreeBSD buildbots are so unreliable that they don't get much attention
>> when they go red"
>>
>> Stable reliable buildbots and a few more volunteers more familiar with
>> BSDs might be a great and welcome addition to python-dev.
>>
>> As for getting this working - I would love a patch. You are going to
>> want to start with python-trunk and look in setup.py. You are going to
>> want to adjust the flags the package uses:
>>
>> elif platform in ('freebsd5', 'freebsd6', 'freebsd7', 'freebsd8'):
>> # FreeBSD's P1003.1b semaphore support is very experimental
>> # and has many known problems. (as of June 200
>> macros = dict( # FreeBSD
>> HAVE_SEM_OPEN=0,
>> HAVE_SEM_TIMEDWAIT=0,
>> HAVE_FD_TRANSFER=1,
>> )
>> libraries = []
>>
>> You will also need to look at: Lib/multiprocessing/synchronize.py to
>> disable the import error - Modules/_multiprocessing/multiprocessing.h
>> will need to be updated for the proper ifdefs for the bsd(s) as well.
>> Finally, the core of the semaphore usage is in
>> Modules/_multiprocessing/semaphore.c
>>
>> I apologize we/I could not get this in for 2.6
>>
>> -jesse

>
> This is exactly the sort of response I was hoping for. Thanks for the
> additional background on the problem. I'll take a look at the code and
> see if I can figure out a patch. I'll also read up on the buildbot
> issue ( I think I saw a link about that)...I might have a stable co-
> located FreeBSD box that could be used.
> -Alan
> --
> http://mail.python.org/mailman/listinfo/python-list
>


I know I really appreciate the additional set of eyes on this, so thanks Alan!
 
Reply With Quote
 
Philip Semanchuk
Guest
Posts: n/a
 
      10-22-2008

On Oct 22, 2008, at 10:11 AM, Jesse Noller wrote:

> On Tue, Oct 21, 2008 at 6:45 PM, <(E-Mail Removed)> wrote:
>> It seems that the multiprocessing module in 2.6 is broken for *BSD;
>> I've seen issue 3770 regarding this. I'm curious if there are more
>> details on this issue since the posts in 3770 were a bit unclear. For
>> example, one post claimed that the problem was that sem_open isn't
>> implemented in *BSD, but it is available on FreeBSD 7 (I checked).
>> I'd
>> be willing to help get this working if someone could point me in the
>> right direction.
>> --
>> http://mail.python.org/mailman/listinfo/python-list
>>

>
> The BSD issue was raised late in the cycle for 2.6. The problem is
> that FBSD's support is "very experimental" as Phillip points out - and
> OpenBSD doesn't even have them.
>
> Due to the lateness of the issue and a finite amount of time I have to
> work on things, I chose to disable support for this on the various
> *BSDs until I can cook up a stable patch or have one provided by
> someone more familiar with the inner workings of Free-BSD. OpenBSD
> support is a non-starter.


Hi Jesse,
I wasn't aware of the multiprocessing module. It looks slick! Well done.

I don't know if you clicked on the link I gave for my posix_ipc
module, but it looks like we're duplicating effort to some degree. My
module makes POSIX semaphore & shared memory primitives available to
Python programs. Obviously, what you've done is much more sophisticated.

One oversight I noticed the multiprocessing module docs is that a
semaphore's acquire() method shouldn't have a timeout on OS X as
sem_timedwait() isn't supported on that platform. (You note OS X's
lack of support for sem_getvalue() elsewhere.)

A question - how do you handle the difference in error messages on
various platforms? For instance, sem_trywait() raises error 35,
"Resource temporarily unavailable" under OS X but error 11 under
Ubuntu. Right now I'm just passing these up to the (Python) caller as
OSErrors. This makes it really hard for the Python programmer to write
cross-platform code.

The only solution I can think of (which I haven't coded yet) is to
compile & run a series of small C programs during setup.py that test
things like sem_trywait() to see what errors occur, and provide those
constants to my main .c module so that it can detect those errors
exactly and wrap them into a specific, custom error for the Python
caller.

Any thoughts on this?

Cheers
Philip



 
Reply With Quote
 
Jesse Noller
Guest
Posts: n/a
 
      10-22-2008
On Wed, Oct 22, 2008 at 11:06 AM, Philip Semanchuk <(E-Mail Removed)> wrote:
>> The BSD issue was raised late in the cycle for 2.6. The problem is
>> that FBSD's support is "very experimental" as Phillip points out - and
>> OpenBSD doesn't even have them.
>>
>> Due to the lateness of the issue and a finite amount of time I have to
>> work on things, I chose to disable support for this on the various
>> *BSDs until I can cook up a stable patch or have one provided by
>> someone more familiar with the inner workings of Free-BSD. OpenBSD
>> support is a non-starter.

>
> Hi Jesse,
> I wasn't aware of the multiprocessing module. It looks slick! Well done.
>


The credit goes to R. Oudkerk, the original author of the pyprocessing
library - I'm simply a rabid user who managed to wrangle it into
Python-Core. See: http://www.python.org/dev/peps/pep-0371/

> I don't know if you clicked on the link I gave for my posix_ipc module, but
> it looks like we're duplicating effort to some degree. My module makes POSIX
> semaphore & shared memory primitives available to Python programs.
> Obviously, what you've done is much more sophisticated.
>


I actually saw your stuff cross the 'tubes - it looks darned nice as a
lower-level interface. What the MP package is meant to be is obviously
much more high level (and "thread like"), MP goes out of it's way to
hide the gritty internals of the semaphore management/etc - posix_ipc
is much more low level than that.

> One oversight I noticed the multiprocessing module docs is that a
> semaphore's acquire() method shouldn't have a timeout on OS X as
> sem_timedwait() isn't supported on that platform. (You note OS X's lack of
> support for sem_getvalue() elsewhere.)


Please file a ticket or update http://bugs.python.org/issue4012 so I
don't loose it, my memory is increasingly lossy. Good catch.

>
> A question - how do you handle the difference in error messages on various
> platforms? For instance, sem_trywait() raises error 35, "Resource
> temporarily unavailable" under OS X but error 11 under Ubuntu. Right now I'm
> just passing these up to the (Python) caller as OSErrors. This makes it
> really hard for the Python programmer to write cross-platform code.
>


If you look at the code, we're pretty much raising OSError - it's
possible we could enhance this in later versions, but given MP is
supposed to be a cross-platform as possible and protect the user from
the seedy underbelly of semaphores/pipes/etc - when an OSError does
occur, it's generally a bug in our code, not the users.

> The only solution I can think of (which I haven't coded yet) is to compile &
> run a series of small C programs during setup.py that test things like
> sem_trywait() to see what errors occur, and provide those constants to my
> main .c module so that it can detect those errors exactly and wrap them into
> a specific, custom error for the Python caller.
>
> Any thoughts on this?
>


That's actually (while feeling hacky) a possibly sensible idea, the
problem is is that you'd need to maintain documentation to tell users
the exceptions for their platform.

-jesse
 
Reply With Quote
 
Philip Semanchuk
Guest
Posts: n/a
 
      10-22-2008

On Oct 22, 2008, at 11:37 AM, Jesse Noller wrote:

> On Wed, Oct 22, 2008 at 11:06 AM, Philip Semanchuk <(E-Mail Removed)
> > wrote:
>>> One oversight I noticed the multiprocessing module docs is that a

>> semaphore's acquire() method shouldn't have a timeout on OS X as
>> sem_timedwait() isn't supported on that platform. (You note OS X's
>> lack of
>> support for sem_getvalue() elsewhere.)

>
> Please file a ticket or update http://bugs.python.org/issue4012 so I
> don't loose it, my memory is increasingly lossy. Good catch.


I updated issue4012.

>> A question - how do you handle the difference in error messages on
>> various
>> platforms? For instance, sem_trywait() raises error 35, "Resource
>> temporarily unavailable" under OS X but error 11 under Ubuntu.
>> Right now I'm
>> just passing these up to the (Python) caller as OSErrors. This
>> makes it
>> really hard for the Python programmer to write cross-platform code.
>>

>
> If you look at the code, we're pretty much raising OSError - it's
> possible we could enhance this in later versions, but given MP is
> supposed to be a cross-platform as possible and protect the user from
> the seedy underbelly of semaphores/pipes/etc - when an OSError does
> occur, it's generally a bug in our code, not the users.


Gotcha. I had a look at the code, and you're testing for errno ==
EAGAIN when sem_trywait() fails. This is correct for OS X and Ubuntu
(the platforms I mentioned above) and probably most other Unices. I
just don't have confidence that it will be true across all platforms,
esp. ones to which I don't have access like AIX and big-iron systems
that support POSIX. Maybe I am just taking defensive programming too
far.

>> The only solution I can think of (which I haven't coded yet) is to
>> compile &
>> run a series of small C programs during setup.py that test things
>> like
>> sem_trywait() to see what errors occur, and provide those constants
>> to my
>> main .c module so that it can detect those errors exactly and wrap
>> them into
>> a specific, custom error for the Python caller.
>>
>> Any thoughts on this?
>>

>
> That's actually (while feeling hacky) a possibly sensible idea, the
> problem is is that you'd need to maintain documentation to tell users
> the exceptions for their platform.


If I pass all errors up as OSError, yes, that will be true. But that's
actually the situation I was trying to avoid.

By generating the error at install time, I can see exactly what
platform X returns in that situation (e.g. 11). Then I can create a
#define something like this:
#define ERRNO_WHEN_CALLING_SEM_TRYWAIT_ON_A_LOCKED_SEMAPHO RE 11

My main .c module can then test for errno ==
ERRNO_WHEN_CALLING_SEM_TRYWAIT_ON_A_LOCKED_SEMAPHO RE and wrap that in
a custom error, like posix_ipc.SemaphoreBusyError.


Again, maybe I'm just taking defensive programming too far. I've been
bitten by other cross-platform inconsistencies at the C API level and
I'm trying to get the jump on them here.

Cheers
Philip


 
Reply With Quote
 
MRAB
Guest
Posts: n/a
 
      10-23-2008
On Oct 22, 5:14*pm, Philip Semanchuk <(E-Mail Removed)> wrote:
> On Oct 22, 2008, at 11:37 AM, Jesse Noller wrote:
>
> > On Wed, Oct 22, 2008 at 11:06 AM, Philip Semanchuk <(E-Mail Removed)
> > > wrote:
> >>> One oversight I noticed the multiprocessing module docs is that a
> >> semaphore's acquire() method shouldn't have a timeout on OS X as
> >> sem_timedwait() isn't supported on that platform. (You note OS X's *
> >> lack of
> >> support for sem_getvalue() elsewhere.)

>
> > Please file a ticket or updatehttp://bugs.python.org/issue4012so I
> > don't loose it, my memory is increasingly lossy. Good catch.

>
> I updated issue4012.
>
> >> A question - how do you handle the difference in error messages on *
> >> various
> >> platforms? For instance, sem_trywait() raises error 35, "Resource
> >> temporarily unavailable" under OS X but error 11 under Ubuntu. *
> >> Right now I'm
> >> just passing these up to the (Python) caller as OSErrors. This *
> >> makes it
> >> really hard for the Python programmer to write cross-platform code.

>
> > If you look at the code, we're pretty much raising OSError - it's
> > possible we could enhance this in later versions, but given MP is
> > supposed to be a cross-platform as possible and protect the user from
> > the seedy underbelly of semaphores/pipes/etc - when an OSError does
> > occur, it's generally a bug in our code, not the users.

>
> Gotcha. I had a look at the code, and you're testing for errno == *
> EAGAIN when sem_trywait() fails. This is correct for OS X and Ubuntu *
> (the platforms I mentioned above) and probably most other Unices. I *
> just don't have confidence that it will be true across all platforms, *
> esp. ones to which I don't have access like AIX and big-iron systems *
> that support POSIX. Maybe I am just taking defensive programming too *
> far.
>
>
>
> >> The only solution I can think of (which I haven't coded yet) is to *
> >> compile &
> >> run a series of small C programs during setup.py that test things *
> >> like
> >> sem_trywait() to see what errors occur, and provide those constants *
> >> to my
> >> main .c module so that it can detect those errors exactly and wrap *
> >> them into
> >> a specific, custom error for the Python caller.

>
> >> Any thoughts on this?

>
> > That's actually (while feeling hacky) a possibly sensible idea, the
> > problem is is that you'd need to maintain documentation to tell users
> > the exceptions for their platform.

>
> If I pass all errors up as OSError, yes, that will be true. But that's *
> actually the situation I was trying to avoid.
>
> By generating the error at install time, I can see exactly what *
> platform X returns in that situation (e.g. 11). Then I can create a *
> #define something like this:
> #define ERRNO_WHEN_CALLING_SEM_TRYWAIT_ON_A_LOCKED_SEMAPHO RE *11
>
> My main .c module can then test for errno == *
> ERRNO_WHEN_CALLING_SEM_TRYWAIT_ON_A_LOCKED_SEMAPHO RE and wrap that in *
> a custom error, like posix_ipc.SemaphoreBusyError.
>
> Again, maybe I'm just taking defensive programming too far. I've been *
> bitten by other cross-platform inconsistencies at the C API level and *
> I'm trying to get the jump on them here.
>

Re the error codes, the OS-specific code can be mapped to an OS-
agnostic(?) code. On Windows there's a Windows-specific subclass of
OSError, WindowsError, containing the Windows error code winerror as
well as errno which is inherited from OSError.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
multiprocessing module and os.close(sys.stdin.fileno()) Graham Dumpleton Python 4 02-22-2009 05:41 AM
Distributing Python Apps on Linux\BSD PurpleServerMonkey Python 3 03-22-2008 02:54 PM
Does ANYONE use Pmw and BLT on Linux/BSD? j_mckitrick Python 0 05-14-2004 02:15 AM
F5 BigIP and 3DNS are running on BSD/OS and BSD/OS is discontinued Baby Peanut Cisco 0 09-11-2003 12:55 AM
Python signal delivery under BSD 4.4 Elf M. Sternberg Python 8 08-16-2003 03:29 AM



Advertisments