Interrput a thread
 

Hi all, I need to stop a threaded (using CTR+C or kill) application if
it runs too much or if I decide to resume the work later.
I come up with the following test implementation but I wanted some
suggestion from you on how I can implement what I need in a better or
more pythonic way. Here the code:

import os
import signal
import time
from threading import Thread, current_thread
from queue import LifoQueue, Empty

COMMAND = {"STOP": 0, "NORMAL": 1}
THREAD_NUM = 5

lq = LifoQueue()

print("{0}\n".format(os.getpid()))

class InterceptInterrupt(Exception):
pass

class Handler:
def __init__(self, queue):
self._queue = queue
def __del__(self):
print("Bye bye!")
def getHandler(self, signum, frame):
print("Interrupt raised!")
for _ in range(THREAD_NUM):
self._queue.put((COMMAND["STOP"], None))
raise InterceptInterrupt

h = Handler(lq)

signal.signal(signal.SIGINT, h.getHandler)

for i in range(25):
lq.put((COMMAND["NORMAL"], i))

def do_work(queue):
while True:
time.sleep(5)
try:
cmd, value = queue.get(block=False)
if cmd == COMMAND["STOP"]:
print("{0}: STOP command
received!".format(current_thread().name))
break
elif cmd == COMMAND["NORMAL"]:
print(value)
except Empty:
break

threads = [Thread(target=do_work, args=(lq,)) for _ in
range(THREAD_NUM)]

for t in threads:
t.start()

while not lq.empty():
try:
time.sleep(1)
except (IOError, InterceptInterrupt):
break

for t in threads:
t.join()

if lq.empty():
print("The queue is empty.")
else:
print("The queue is NOT empty. Some additional work has to be
done.")

Thank you,
Mattia

Re: Interrput a thread
 

On 12/29/2010 3:31 PM, gervaz wrote:
> Hi all, I need to stop a threaded (using CTR+C or kill) application if
> it runs too much or if I decide to resume the work later.
> I come up with the following test implementation but I wanted some
> suggestion from you on how I can implement what I need in a better or
> more pythonic way.

You can't send signals to individual threads under CPython.
Even where the OS supports it, CPython does not. See
"http://docs.python.org/library/signal.html"

Nor can you kill a thread. All you can do is ask it
nicely to stop itself.

Even worse, sending control-C to a multi-thread program
is unreliable in CPython. See "http://blip.tv/file/2232410"
for why. It's painful.

John Nagle

Re: Interrput a thread
 

On 2010-12-31 10:28:26 -0800, John Nagle said:

> Even worse, sending control-C to a multi-thread program
> is unreliable in CPython. See "http://blip.tv/file/2232410"
> for why. It's painful.

AFIK, that has been resolved in Python 3.2 with the introduction of an
intelligent thread scheduler as part of the GIL release/acquire process.

- Alice.

Re: Interrput a thread
 

On 31 Dic 2010, 23:25, Alice Bevan–McGregor <al...@gothcandy.com>
wrote:
> On 2010-12-31 10:28:26 -0800, John Nagle said:
>
> > Even worse, sending control-C to a multi-thread program
> > is unreliable in CPython. *See "http://blip.tv/file/2232410"
> > for why. *It's painful.
>
> AFIK, that has been resolved in Python 3.2 with the introduction of an
> intelligent thread scheduler as part of the GIL release/acquire process.
>
> * * * * - Alice.

Ok, but then suppose I have multiple long running threads that I want
to delete/suspend because they are tooking too much time, which
solution do you propose?

Thanks,

Mattia

Re: Interrput a thread
 

gervaz <gervaz@gmail.com> writes:

> On 31 Dic 2010, 23:25, Alice Bevan–McGregor <al...@gothcandy.com>
> wrote:
>> On 2010-12-31 10:28:26 -0800, John Nagle said:
>>
>> > Even worse, sending control-C to a multi-thread program
>> > is unreliable in CPython. Â*See "http://blip.tv/file/2232410"
>> > for why. Â*It's painful.
>>
>> AFIK, that has been resolved in Python 3.2 with the introduction of an
>> intelligent thread scheduler as part of the GIL release/acquire process.
>>
>> Â* Â* Â* Â* - Alice.
>
> Ok, but then suppose I have multiple long running threads that I want
> to delete/suspend because they are tooking too much time, which
> solution do you propose?

If possible, use multiple processes instead.

Diez

Re: Interrput a thread
 

On 3 Gen, 17:47, de...@web.de (Diez B. Roggisch) wrote:
> gervaz <ger...@gmail.com> writes:
> > On 31 Dic 2010, 23:25, Alice Bevan–McGregor <al...@gothcandy.com>
> > wrote:
> >> On 2010-12-31 10:28:26 -0800, John Nagle said:
>
> >> > Even worse, sending control-C to a multi-thread program
> >> > is unreliable in CPython. *See "http://blip.tv/file/2232410"
> >> > for why. *It's painful.
>
> >> AFIK, that has been resolved in Python 3.2 with the introduction of an
> >> intelligent thread scheduler as part of the GIL release/acquire process.
>
> >> * * * * - Alice.
>
> > Ok, but then suppose I have multiple long running threads that I want
> > to delete/suspend because they are tooking too much time, which
> > solution do you propose?
>
> If possible, use multiple processes instead.
>
> Diez- Nascondi testo citato
>
> - Mostra testo citato -

Multiple processes, ok, but then regarding processes' interruption
there will be the same problems pointed out by using threads?

Mattia

Re: Interrput a thread
 

On Jan 3, 3:22*pm, gervaz <ger...@gmail.com> wrote:
> On 3 Gen, 17:47, de...@web.de (Diez B. Roggisch) wrote:
>
>
>
> > gervaz <ger...@gmail.com> writes:
> > > On 31 Dic 2010, 23:25, Alice Bevan–McGregor <al...@gothcandy.com>
> > > wrote:
> > >> On 2010-12-31 10:28:26 -0800, John Nagle said:
>
> > >> > Even worse, sending control-C to a multi-thread program
> > >> > is unreliable in CPython. *See "http://blip.tv/file/2232410"
> > >> > for why. *It's painful.
>
> > >> AFIK, that has been resolved in Python 3.2 with the introduction of an
> > >> intelligent thread scheduler as part of the GIL release/acquire process.
>
> > >> * * * * - Alice.
>
> > > Ok, but then suppose I have multiple long running threads that I want
> > > to delete/suspend because they are tooking too much time, which
> > > solution do you propose?
>
> > If possible, use multiple processes instead.
>
> > Diez- Nascondi testo citato
>
> > - Mostra testo citato -
>
> Multiple processes, ok, but then regarding processes' interruption
> there will be the same problems pointed out by using threads?
>

No. Processes can be terminated easily on all major platforms. See
`os.kill`.

Jean-Paul

Re: Interrput a thread
 

On Jan 3, 4:06*pm, Jean-Paul Calderone <calderone.jeanp...@gmail.com>
wrote:
>
> > Multiple processes, ok, but then regarding processes' interruption
> > there will be the same problems pointed out by using threads?
>
> No. *Processes can be terminated easily on all major platforms. *See
> `os.kill`.
>

Yes, but that's not the whole story, now is it? It's certainly much
more reliable and easier to kill a process. It's not any easier to do
it and retain defined behavior, depending on exactly what you're
doing. For example, if you kill it while it's in the middle of
updating shared memory, you can potentially incur undefined behavior
on the part of any process that can also access shared memory.

In short, taking a program that uses threads and shared state and
simply replacing the threads with processes will likely not gain you a
thing. It entirely depends on what those threads are doing and how
they do it.

Adam

Re: Interrput a thread
 

On 3 Gen, 22:17, Adam Skutt <ask...@gmail.com> wrote:
> On Jan 3, 4:06*pm, Jean-Paul Calderone <calderone.jeanp...@gmail.com>
> wrote:
>
>
>
> > > Multiple processes, ok, but then regarding processes' interruption
> > > there will be the same problems pointed out by using threads?
>
> > No. *Processes can be terminated easily on all major platforms. *See
> > `os.kill`.
>
> Yes, but that's not the whole story, now is it? *It's certainly much
> more reliable and easier to kill a process. *It's not any easier to do
> it and retain defined behavior, depending on exactly what you're
> doing. *For example, if you kill it while it's in the middle of
> updating shared memory, you can potentially incur undefined behavior
> on the part of any process that can also access shared memory.
>
> In short, taking a program that uses threads and shared state and
> simply replacing the threads with processes will likely not gain you a
> thing. *It entirely depends on what those threads are doing and how
> they do it.
>
> Adam

As per the py3.1 documentation, os.kill is only available in the Unix
os. Regarding the case pointed out by Adam I think the best way to
deal with it is to create a critical section so that the shared memory
will be updated in an atomic fashion. Btw it would be useful to take a
look at some actual code/documentation in order to understand how
others dealt with the problem...

Ciao,

Mattia

Re: Interrput a thread
 

On Jan 3, 4:17*pm, Adam Skutt <ask...@gmail.com> wrote:
> On Jan 3, 4:06*pm, Jean-Paul Calderone <calderone.jeanp...@gmail.com>
> wrote:
>
>
>
> > > Multiple processes, ok, but then regarding processes' interruption
> > > there will be the same problems pointed out by using threads?
>
> > No. *Processes can be terminated easily on all major platforms. *See
> > `os.kill`.
>
> Yes, but that's not the whole story, now is it? *It's certainly much
> more reliable and easier to kill a process. *It's not any easier to do
> it and retain defined behavior, depending on exactly what you're
> doing. *For example, if you kill it while it's in the middle of
> updating shared memory, you can potentially incur undefined behavior
> on the part of any process that can also access shared memory.

Then don't use shared memory.

>
> In short, taking a program that uses threads and shared state and
> simply replacing the threads with processes will likely not gain you a
> thing. *It entirely depends on what those threads are doing and how
> they do it.
>

Of course. The whole point here is not about threads vs processes.
It's about shared memory concurrency vs non-shared memory
concurrency. You can implement both with threads and both with
processes, but threads are geared towards shared memory and processes
are geared towards non-shared memory. So what most people mean by
"use processes" is "don't use shared memory".

Jean-Paul