«

On Fri, 2010-10-29 at 08:39 -0400, Neal Becker wrote:
> Adam Tauno Williams wrote:
>
> > On Fri, 2010-10-29 at 08:12 -0400, Neal Becker wrote:
> >> Seems multiprocessing doesn't behave well with signals:
> >> ---------
> >> from multiprocessing import Pool
> >> import time
> >> def sleep (dummy):
> >> time.sleep (10)
> >> if __name__ == '__main__':
> >> pool = Pool (processes=2)
> >> result = pool.map (sleep, range (4))
> >> -------------
> >> start it up
> >> $ python test_multip.py
> >> ----------------------
> >> ps auxf | grep python
> >> nbecker 6605 1.6 0.1 338192 6952 pts/1 Sl+ 08:03 0:00 |
> >> \_ python test_multip.py
> >> nbecker 6606 0.0 0.1 186368 4760 pts/1 S+ 08:03 0:00 |
> >> \_ python test_multip.py
> >> nbecker 6607 0.0 0.1 186372 4740 pts/1 S+ 08:03 0:00 |
> >> \_ python test_multip.py
> >> kill 6607
> >> ps auxf | grep python
> >> nbecker 6605 0.5 0.1 338192 6952 pts/1 Sl+ 08:03 0:00 |
> >> \_ python test_multip.py
> >> nbecker 6606 0.0 0.1 186368 4760 pts/1 S+ 08:03 0:00 |
> >> \_ python test_multip.py
> >> nbecker 6607 0.0 0.0 0 0 pts/1 Z+ 08:03 0:00 |
> >> \_ [python] <defunct>
> >> kill 6606
> >> ps auxf | grep python
> >> nbecker 6605 0.3 0.1 338192 6952 pts/1 Sl+ 08:03 0:00 |
> >> \_ python test_multip.py
> >> nbecker 6606 0.0 0.0 0 0 pts/1 Z+ 08:03 0:00 |
> >> \_ [python] <defunct>
> >> nbecker 6607 0.0 0.0 0 0 pts/1 Z+ 08:03 0:00 |
> >> \_ [python] <defunct>
> >> Now we have 2 dead children and the parent is hung forever.
> >> Isn't this a serious defect?
> > No, I think this is just POSIX/UNIX process behavior. If the parent
> > never joins on the child the child can never exit [which is what a
> > Zombie process is].
> > For example, see the do_verify_workers method in
> <http://coils.hg.sourceforge.net/hgweb/coils/coils/file/6ab5ade3e488/src/coils/logic/workflow/services/executor.py>
> > A parent process needs to make some effort to reap its children.
> Yes, and isn't this a defect in mulitprocessing module that the parent
> process does not reap its children in response to signals like show above?

No, I don't think so. You're asking the module to over generalize
behavior. Reaping of the child is important, and that the child needs
to be reaped may matter to the master child (why? did something go
wrong?). Silently reaping them [which would reduce the size of the
Pool? Or would it dynamically create a new worker?] might have
unintended side effects. Maybe since Pool specifically generalizes
child management you could make an argument it should reap, but I'm not
sure. Personally I'd recommend that your worker processes include a
signal handler to do something smart in the case of a "-15" [for which
there isn't really a thread equivalent - can you sent a SystemV style
signal to an individual thread in a process? I don't think so.]

How would a 'traditional' thread pool behave if a thread abended? [of
course, that depends on the thread-pool implementation] The correct
behavior in case of an exception in a thread is a topic of some debate.

On Oct 29, 10:08*am, Adam Tauno Williams <awill...@whitemice.org>
wrote:
> signal handler to do something smart in the case of a "-15" [for which
> there isn't really a thread equivalent - can you sent a SystemV style
> signal to an individual thread in a process? *I don't think so.]

Yes. pthread_kill(P)

Jean-Paul

On Fri, 2010-10-29 at 07:31 -0700, Jean-Paul Calderone wrote:
> On Oct 29, 10:08 am, Adam Tauno Williams <awill...@whitemice.org>
> wrote:
> > signal handler to do something smart in the case of a "-15" [for which
> > there isn't really a thread equivalent - can you sent a SystemV style
> > signal to an individual thread in a process? I don't think so.]
> Yes. pthread_kill(P)

Eh, only sort of.

<quote>
Signal dispositions are process-wide: if a signal handler is installed,
the handler will be invoked in the thread thread, but if the disposition
of the signal is "stop", "continue", or "terminate", this action will
affect the whole process.
</quote>

Also phthread_kill can only be called for a thread in the same process.
Which is quite different that SystemV signals.

So there can't really be an exact 1:1 match of how thread-signaling vs.
multiprocessing-process-signaling works; which was my point.

On Oct 29, 10:08*am, Adam Tauno Williams <awill...@whitemice.org>
wrote:
> No, I don't think so. *You're asking the module to over generalize
> behavior. *Reaping of the child is important, and that the child needs
> to be reaped may matter to the master child (why? did something go
> wrong?).
And such information is made available when you reap the child
process. There are a ton of ways for the module to provide sane and
proper default behavior and still provide hooks for people who need to
modify the behavior or retrieve information. However, I can't imagine
a use case where the end-developer cares about the process' exit
status from a multiprocess.Pool.

> Silently reaping them [which would reduce the size of the
> Pool? Or would it dynamically create a new worker?] might have
> unintended side effects.
Such as? Personally, even if you do enumerate one, I'm not sure how
much it matters, as an unintended side effect has already occurred.
It's too late to avoid side effects, they've already happened.
Moreover, merely blithely continuing on invites all sorts of fun side
effects, like the lovely "undefined behavior" side effect.

> How would a 'traditional' thread pool behave if a thread abended? *[of
> course, that depends on the thread-pool implementation] *The correct
> behavior in case of an exception in a thread is a topic of some debate.
Yes, but we're not talking about an exception in a thread, we're
talking about a thread being terminated by the OS. Which should be
treated like every other OS termination: as a fatal error. There's
not way for the multiprocessing module to fully recover from such an
event to a known state, so the only sensible thing left is
termination. Anything else runs the risk of undefined behavior.

Adam