Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > KeyboardInterrupt and threading

Reply
Thread Tools

KeyboardInterrupt and threading

 
 
Ivan Nestlerode
Guest
Posts: n/a
 
      01-02-2004
Hello comp.lang.python,

I am attempting to write a threaded program right now in Python
(version 2.3 on Debian unstable) and the current behavior of
KeyboardInterrupt is causing me grief. From the documentation of the
thread module in Python 2.3.3:

"Threads interact strangely with interrupts: the KeyboardInterrupt
exception will be received by an arbitrary thread. (When the signal
module is available, interrupts always go to the main thread.)"

So the current behavior is that some "arbitrarily chosen" thread
receives the KeyboardInterrupt exception when the user hits Ctrl-C. My
question is whether there is some way to always make the main thread
receive the KeyboardInterrupt. The snippet at the end of the
documentation is vague/confusing. It sounds like KeyboardInterrupt
would always be delivered to the main thread, but I have tested this
and it is definitely not behaving that way. Maybe it just means that
the Unix signal is received by the main thread's signal handler (and
at that point an arbitrary thread is picked for KeyboardInterrupt). I
don't know.

The reason I cannot use the current "arbitrary thread"
KeyboardInterrupt delivery model:

The current model seems to deliver the KeyboardInterrupt to the
busiest thread (i.e. some sub-thread doing lots of computation, as
opposed to my main thread that is sitting in a sleep() loop). This
causes problems because the busy thread that I spawn runs library code
that was not written by me. That library code may contain lines like
"except:" that mask the KeyboardInterrupt so that the user experience
is that the app seems to ignore Ctrl-C.

If I could guarantee delivery of KeyboardInterrupt to the main thread,
I could control things much more easily/consistently. With the current
semantics code reuse becomes very difficult (I cannot use any code
that is "sloppy" with its exception catching or Ctrl-C will be
ignored).

Any ideas (besides rewriting the sloppy library code) would be much
appreciated.

Thanks,
-Ivan
 
Reply With Quote
 
 
 
 
Paul McGuire
Guest
Posts: n/a
 
      01-03-2004
"Ivan Nestlerode" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed) om...
> Hello comp.lang.python,
>
> I am attempting to write a threaded program right now in Python
> (version 2.3 on Debian unstable) and the current behavior of
> KeyboardInterrupt is causing me grief. From the documentation of the
> thread module in Python 2.3.3:
>
> "Threads interact strangely with interrupts: the KeyboardInterrupt
> exception will be received by an arbitrary thread. (When the signal
> module is available, interrupts always go to the main thread.)"
>

Is the signal module not available to you?
<snip>
> Any ideas (besides rewriting the sloppy library code) would be much
> appreciated.
>
> Thanks,
> -Ivan


Assuming you have access to the signal module:
1. Create a boolean that you can periodically test for in your thread code,
to see if you should keep processing. This may be a global singleton, a
global boolean, or something, but it needs to be visible to both the main
thread and the threaded code. You probably have some compute intensive
loops such as:
while( prime_factors < 1e15 ):
or
while( optimization != converged ):
or even just
while( True ):

Change these to reference the "keep processing" flag.
while( keep_processing and prime_factors < 1e15 ):
or
while( keep_processing and optimization != converged ):
or even just
while( keep_processing ): # no need to 'and' with True

2. Create a signal handler to trap for SIGINT. In the signal handler, set
the "keep-processing" bit to False:

import signal

def discontinue_processing(signl, frme):
global keep_processing
keep_processing = False
return 0

signal.signal( signal.SIGINT, discontinue_processing )


Now your main thread will get the ^C interrupt and clear the
"keep_processing" flag, which will eventually get picked up by your worker
thread the next time it tests for it.

HTH,
-- Paul


 
Reply With Quote
 
 
 
 
Michael Hudson
Guest
Posts: n/a
 
      01-05-2004
http://www.velocityreviews.com/forums/(E-Mail Removed) (Ivan Nestlerode) writes:

> Hello comp.lang.python,
>
> I am attempting to write a threaded program right now in Python
> (version 2.3 on Debian unstable) and the current behavior of
> KeyboardInterrupt is causing me grief. From the documentation of the
> thread module in Python 2.3.3:
>
> "Threads interact strangely with interrupts: the KeyboardInterrupt
> exception will be received by an arbitrary thread. (When the signal
> module is available, interrupts always go to the main thread.)"


On debian, the signal module certainly should be available, and the
KeyboardInterrupt exception should go to the main thread. Or at
least, that's what I thought. You're definitely seeing it being
delivered to an arbitrary thread?

Cheers,
mwh

--
The Programmer's Quick Guide To Python (Time Machine version):
You try to shoot yourself in the foot, only to realize that
there's no need, since Guido thoughtfully shot you in the foot
years ago. -- Nick Mathewson, comp.lang.python
 
Reply With Quote
 
Ivan Nestlerode
Guest
Posts: n/a
 
      01-07-2004
Michael Hudson <(E-Mail Removed)> wrote in message news:<(E-Mail Removed)>...
> On debian, the signal module certainly should be available, and the
> KeyboardInterrupt exception should go to the main thread. Or at
> least, that's what I thought. You're definitely seeing it being
> delivered to an arbitrary thread?
>
> Cheers,
> mwh


After a closer look, I am not seeing KeyboardInterrupt being delivered
to non-main threads. What I am seeing is that Ctrl-C does not work at
all when non-main threads are CPU intensive. I had previously jumped
to the conclusion that because Ctrl-C didn't work and because the
non-main thread was sloppy ("except:"), that Ctrl-C wasn't working
because KeyboardInterrupt was going to the sub-thread. That was an
incorrect conclusion.

So I guess the entire problem can be restated as:
How do I make Ctrl-C work properly when the non-main thread is busier
than the main thread?

The short program at the end of this post demonstrates the problem
with Ctrl-C and busy non-main threads. The idea behind this script is
that the main thread launches a busy sub-thread. Ctrl-C is supposed to
be caught in the main thread, stopping both threads, and ending the
program (with some printouts). As written, the program does not
respond to Ctrl-C even if I hold it down continuously. If you
uncomment the print statement in the main thread, all of the sudden it
starts to work. Why?

I think it has something to do with the main thread needing some
larger amount of CPU to detect the signal and the KeyboardInterrupt. I
think this is a bug though (I shouldn't have to put hacks into the
main thread to get this to work).

Any thoughts?

Thanks,
-Ivan


#!/usr/bin/python

from threading import Event, Thread
from time import sleep

class CpuHogThread(Thread):
def __init__(self):
Thread.__init__(self)
self.counter = 0
self.stopEvent = Event()

def join(self):
self.stopEvent.set()
Thread.join(self)

def run(self):
try:
while not self.stopEvent.isSet():
self.counter += 1
except KeyboardInterrupt:
print 'CPU hog subthread caught KeyboardInterrupt'

if __name__ == '__main__':
t = CpuHogThread()
t.start()
print 'CPU hog subthread spawned'
try:
while True:
sleep(2)
# without the next print, Ctrl-C is completely ignored
#print 'still sleeping'
except KeyboardInterrupt:
print 'main thread caught KeyboardInterrupt'
t.join()
print 'CPU hog subthread joined at counter %d' % t.counter
 
Reply With Quote
 
Ivan Nestlerode
Guest
Posts: n/a
 
      01-07-2004
"Paul McGuire" <(E-Mail Removed)> wrote in message news:<XhEJb.40848$(E-Mail Removed)>...
> Is the signal module not available to you?
> <snip>
> Assuming you have access to the signal module:
> 1. Create a boolean that you can periodically test for in your thread code,
> to see if you should keep processing. This may be a global singleton, a
> global boolean, or something, but it needs to be visible to both the main
> thread and the threaded code. You probably have some compute intensive
> loops such as:
> while( prime_factors < 1e15 ):
> or
> while( optimization != converged ):
> or even just
> while( True ):
>
> Change these to reference the "keep processing" flag.
> while( keep_processing and prime_factors < 1e15 ):
> or
> while( keep_processing and optimization != converged ):
> or even just
> while( keep_processing ): # no need to 'and' with True
>
> 2. Create a signal handler to trap for SIGINT. In the signal handler, set
> the "keep-processing" bit to False:
>
> <snip>
>
> Now your main thread will get the ^C interrupt and clear the
> "keep_processing" flag, which will eventually get picked up by your worker
> thread the next time it tests for it.
>
> HTH,
> -- Paul


I do have the signal module (Python 2.3.2 on Linux), but
unfortunately, even registering a signal handler doesn't fix the
problem where busy non-main threads prevent Ctrl-C from working.

Here is another code snippet using a signal handler (this code is very
similar to the code from my second post). If the main thread's loop
has the print statement, everything works fine. If it is commented out
(as written here), Ctrl-C is ignored completely and the program just
keeps running. Why won't this work? Is this a bug?

Thanks,
-Ivan

#!/usr/bin/python

from signal import signal, SIGINT
from threading import Event, Thread
from time import sleep

class CpuHogThread(Thread):
def __init__(self):
Thread.__init__(self)
self.counter = 0
self.stopEvent = Event()

def join(self):
self.stopEvent.set()
Thread.join(self)

def run(self):
try:
while not self.stopEvent.isSet():
self.counter += 1
except KeyboardInterrupt:
print 'CPU hog subthread caught KeyboardInterrupt'

def handler(signal, frme):
stopEvent.set()
print 'main thread SIGINT handler'
t.join()
print 'CPU hog subthread joined at counter %d' % t.counter
return 0

if __name__ == '__main__':
stopEvent = Event()
signal(SIGINT, handler)
t = CpuHogThread()
t.start()

print 'CPU hog subthread spawned'
try:
while not stopEvent.isSet():
sleep(2)
# without the next print, Ctrl-C is completely ignored
#print 'still sleeping'
except KeyboardInterrupt:
print 'main thread caught KeyboardInterrupt'
t.join()
print 'CPU hog subthread joined at counter %d' % t.counter
 
Reply With Quote
 
Jeff Epler
Guest
Posts: n/a
 
      01-07-2004
Running your attached program, it works just fine whereever I try it,
except on a system so old it only has Python 1.5 installed. Even that
worked after a few seconds of editing.

Here's the behavior I saw on Fedora Core 1:
$ python ivan.py
CPU hog subthread spawned
main thread caught KeyboardInterrupt
CPU hog subthread joined at counter 76429

This is the behavior I saw on redhat 9:
$ python /tmp/ivan.py
CPU hog subthread spawned
main thread caught KeyboardInterrupt
CPU hog subthread joined at counter 89897

Here's the behavior I saw on Red Hat 7.2:
CPU hog subthread spawned
main thread caught KeyboardInterrupt
CPU hog subthread joined at counter 31764

Here's the behavior I saw on Windows XP:
C:\...> python ivan.py
CPU hog subthread spawned
main thread caught KeyboardInterrupt
CPU hog subthread joined at counter 665928

Here's the behavior I saw on Red Hat 6.2:
$ python1.5 ivan.py
File "ivan.py", line 19
self.counter += 1
^
SyntaxError: invalid syntax
$ vi ivan.py
$ python1.5 ivan.py
CPU hog subthread spawned
main thread caught KeyboardInterrupt
CPU hog subthread joined at counter 238983

Jeff

 
Reply With Quote
 
Michael Hudson
Guest
Posts: n/a
 
      01-07-2004
(E-Mail Removed) (Ivan Nestlerode) writes:

> Michael Hudson <(E-Mail Removed)> wrote in message news:<(E-Mail Removed)>...
> > On debian, the signal module certainly should be available, and the
> > KeyboardInterrupt exception should go to the main thread. Or at
> > least, that's what I thought. You're definitely seeing it being
> > delivered to an arbitrary thread?
> >
> > Cheers,
> > mwh

>
> After a closer look, I am not seeing KeyboardInterrupt being delivered
> to non-main threads. What I am seeing is that Ctrl-C does not work at
> all when non-main threads are CPU intensive.


Oh...

> I had previously jumped to the conclusion that because Ctrl-C didn't
> work and because the non-main thread was sloppy ("except:"), that
> Ctrl-C wasn't working because KeyboardInterrupt was going to the
> sub-thread. That was an incorrect conclusion.


Well, in a way, good, but perhaps not for you...

> So I guess the entire problem can be restated as: How do I make
> Ctrl-C work properly when the non-main thread is busier than the
> main thread?


This is hard for me to answer -- it works fine for me. What platform
are you on? ISTR Debian, but not versions of anything.

I'm somewhat inclined to start blaming libc at this point...

As you may well be aware, the combination of threads and signals is a
bit of a minefield. Python runs all threads other than the main
thread with all signals masked. My understanding is that when a signal
is delivered to a process it's delivered to an arbitrary thread that
has that signal unmasked -- in Python's case, this should be the main
thread. It kind of sounds like in your case the signal is being
queued up on a non-main thread, but as the signal is never unblocked,
it never gets handled.

[...]
> I think this is a bug though (I shouldn't have to put hacks into the
> main thread to get this to work).


Agreed, but I'm not sure they're in Python.

> Any thoughts?


Upgrade glibc?

Cheers,
mwh

--
ARTHUR: Why are there three of you?
LINTILLAS: Why is there only one of you?
ARTHUR: Er... Could I have notice of that question?
-- The Hitch-Hikers Guide to the Galaxy, Episode 11
 
Reply With Quote
 
Ivan Nestlerode
Guest
Posts: n/a
 
      01-08-2004
Michael Hudson <(E-Mail Removed)> wrote in message news:<(E-Mail Removed)>...
> This is hard for me to answer -- it works fine for me. What platform
> are you on? ISTR Debian, but not versions of anything.
>
> I'm somewhat inclined to start blaming libc at this point...
>
> As you may well be aware, the combination of threads and signals is a
> bit of a minefield. Python runs all threads other than the main
> thread with all signals masked. My understanding is that when a signal
> is delivered to a process it's delivered to an arbitrary thread that
> has that signal unmasked -- in Python's case, this should be the main
> thread. It kind of sounds like in your case the signal is being
> queued up on a non-main thread, but as the signal is never unblocked,
> it never gets handled.
>
> [...]
> > I think this is a bug though (I shouldn't have to put hacks into the
> > main thread to get this to work).

>
> Agreed, but I'm not sure they're in Python.
>
> > Any thoughts?

>
> Upgrade glibc?
>
> Cheers,
> mwh


I've filed the bug with Debian since I think it is something wrong
with their Python or their glibc. The bug is #226547.

FYI, here are the exact details of the kernel/packages that don't seem
to work:

$ uname -a
Linux debian 2.4.23 #1 Sun Nov 30 21:11:49 EST 2003 i686 GNU/Linux

packages and versions:
ii libbz2-1.0 1.0.2-1
ii libc6 2.3.2.ds1-10
ii libdb4.1 4.1.25-10
ii libncurses5 5.3.20030719-4
ii libreadline4 4.3-8
ii libssl0.9.7 0.9.7c-5
ii python 2.3.3-4
ii python2.3 2.3.3-4
ii zlib1g 1:1.2.1-3


-Ivan
 
Reply With Quote
 
Anand Pillai
Guest
Posts: n/a
 
      01-08-2004
There is an easier way than using signals.

Add a keyboard interrupt handler for your main function.
And in the handler code, write a clean up function which
will be able to then manage threads.

This is the method I have used in my HarvestMan program.
I am copying this code here.

------------(copied code)----------------------------------------
try:
self.start_project()
except (KeyboardInterrupt, EOFError):
# Localise links
if not self._cfg.ignorekbinterrupt:
# dont allow to write cache, since it
# screws up existing cache.
GetObject('datamanager').conditional_cache_set()
self.clean_up()
---------------------------------------------------------------------

The clean_up method takes care of cleaning the thread.
There you can add code to pass on the interrupt to the
main thread or perform your own clean up actions.

One way to do this is to use the currentThread() method
of threading module which can tell you whether the thread
is the main or a worker.

One drawback with this method is that you might need to
set your threads as 'daemon' threads to make sure that
the threads dont hang your python program. This is because
the entire python program exits when the only threads left
are daemon threads.

-Anand

(E-Mail Removed) (Ivan Nestlerode) wrote in message news:<(E-Mail Removed). com>...
> Michael Hudson <(E-Mail Removed)> wrote in message news:<(E-Mail Removed)>...
> > This is hard for me to answer -- it works fine for me. What platform
> > are you on? ISTR Debian, but not versions of anything.
> >
> > I'm somewhat inclined to start blaming libc at this point...
> >
> > As you may well be aware, the combination of threads and signals is a
> > bit of a minefield. Python runs all threads other than the main
> > thread with all signals masked. My understanding is that when a signal
> > is delivered to a process it's delivered to an arbitrary thread that
> > has that signal unmasked -- in Python's case, this should be the main
> > thread. It kind of sounds like in your case the signal is being
> > queued up on a non-main thread, but as the signal is never unblocked,
> > it never gets handled.
> >
> > [...]
> > > I think this is a bug though (I shouldn't have to put hacks into the
> > > main thread to get this to work).

> >
> > Agreed, but I'm not sure they're in Python.
> >
> > > Any thoughts?

> >
> > Upgrade glibc?
> >
> > Cheers,
> > mwh

>
> I've filed the bug with Debian since I think it is something wrong
> with their Python or their glibc. The bug is #226547.
>
> FYI, here are the exact details of the kernel/packages that don't seem
> to work:
>
> $ uname -a
> Linux debian 2.4.23 #1 Sun Nov 30 21:11:49 EST 2003 i686 GNU/Linux
>
> packages and versions:
> ii libbz2-1.0 1.0.2-1
> ii libc6 2.3.2.ds1-10
> ii libdb4.1 4.1.25-10
> ii libncurses5 5.3.20030719-4
> ii libreadline4 4.3-8
> ii libssl0.9.7 0.9.7c-5
> ii python 2.3.3-4
> ii python2.3 2.3.3-4
> ii zlib1g 1:1.2.1-3
>
>
> -Ivan

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: FW: [Tutor] Multi-Threading and KeyboardInterrupt Dennis Lee Bieber Python 13 06-16-2009 05:12 AM
FW: [Tutor] Multi-Threading and KeyboardInterrupt Strax-Haber, Matthew (LARC-D320) Python 0 06-11-2009 01:44 PM
Re: exception KeyboardInterrupt and os.system command jepler@unpythonic.net Python 3 11-28-2005 06:21 PM
exception KeyboardInterrupt and os.system command darren kirby Python 1 11-27-2005 11:18 PM
Socket object and KeyboardInterrupt exception PantherSE Python 0 05-16-2005 09:10 PM



Advertisments