Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Simple question about Queue.Queue and threads

Reply
Thread Tools

Simple question about Queue.Queue and threads

 
 
Frank Millman
Guest
Posts: n/a
 
      02-05-2010
Hi all

Assume you have a server process running, a pool of worker threads to
perform tasks, and a Queue.Queue() to pass the tasks to the workers.

In order to shut down the server cleanly, you want to ensure that the
workers have all finished their tasks. I like the technique of putting a
None onto the queue, and have each worker check for None, put None back onto
the queue, and terminate itself.

The main program would look something like this -

q.put(None)
for worker in worker_threads:
worker.join()

At this point you can be sure that each thread has completed its tasks and
terminated itself.

However, the queue is not empty - it still has the final None in it.

Is it advisable to finalise the cleanup like this? -

while not q.empty():
q.get()
q.task_done()
q.join()

Or is this completely redundant?

Thanks

Frank Millman


 
Reply With Quote
 
 
 
 
Steven
Guest
Posts: n/a
 
      02-08-2010
On Feb 5, 7:45*am, "Frank Millman" <(E-Mail Removed)> wrote:
> Hi all
>
> Assume you have a server process running, a pool of worker threads to
> perform tasks, and aQueue.Queue() to pass the tasks to the workers.
>
> In order to shut down the server cleanly, you want to ensure that the
> workers have all finished their tasks. I like the technique of putting a
> None onto thequeue, and have each worker check for None, put None back onto
> thequeue, and terminate itself.
>
> The main program would look something like this -
>
> * * q.put(None)
> * * for worker in worker_threads:
> * * * * worker.join()
>
> At this point you can be sure that each thread has completed its tasks and
> terminated itself.
>
> However, thequeueis not empty - it still has the final None in it.
>
> Is it advisable to finalise the cleanup like this? -
>
> * * while not q.empty():
> * * * * q.get()
> * * * * q.task_done()
> * * q.join()
>
> Or is this completely redundant?
>
> Thanks
>
> Frank Millman


Queue objects have support for this signaling baked in with
q.task_done and q.join.

After the server process has put all tasks into the queue, it can join
the queue itself, not the worker threads.

q.join()

This will block until all tasks have been gotten AND completed. The
worker threads would simply do this:
task_data = q.get()
do_task(task_data)
q.task_done()

Using pairs of get and task_done you no longer need to send a signal.
Just exit the server process and the worker threads will die (assuming
of course, you set .setDaemon(True) before starting each worker
thread).

Steven Rumbalski
 
Reply With Quote
 
 
 
 
Dennis Lee Bieber
Guest
Posts: n/a
 
      02-08-2010
On Mon, 8 Feb 2010 06:51:02 -0800 (PST), Steven
<(E-Mail Removed)> declaimed the following in
gmane.comp.python.general:


>
> Queue objects have support for this signaling baked in with
> q.task_done and q.join.
>

Only in Python 2.5 and later (and I, for one, only upgraded to 2.5
last summer; suspect 2.6 will be chosen later this year). Passing a
unique sentinel works in all versions supporting Queue. And if a second
Queue is used for returns from the threads, each expiring thread could
return its ID for use in a .join() operation -- whereas using the
q.join() method blocks the collector of the return data until all the
tasks are done.

--
Wulfraed Dennis Lee Bieber KD6MOG
http://www.velocityreviews.com/forums/(E-Mail Removed) HTTP://wlfraed.home.netcom.com/

 
Reply With Quote
 
Frank Millman
Guest
Posts: n/a
 
      02-09-2010
On Feb 8, 4:51 pm, Steven <(E-Mail Removed)> wrote:
>
> Queue objects have support for this signaling baked in with
> q.task_done and q.join.
>
> After the server process has put all tasks into the queue, it can join
> the queue itself, not the worker threads.
>
> q.join()
>
> This will block until all tasks have been gotten AND completed. The
> worker threads would simply do this:
> task_data = q.get()
> do_task(task_data)
> q.task_done()
>
> Using pairs of get and task_done you no longer need to send a signal.
> Just exit the server process and the worker threads will die (assuming
> of course, you set .setDaemon(True) before starting each worker
> thread).
>


Thanks, Steven.

This works perfectly in my scenario, and tidies up the code a bit.

Minor point - according to the 2.6 docs, .setDaemon(True) is the old API -
the current way of specifying this is .daemon = True.

Thanks for the tip.

Frank


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Java Threads - Get running threads Pedro Pinto Java 2 04-08-2008 11:44 PM
[new to threads] threads with UI and loop Une bévue Ruby 0 06-14-2006 10:22 AM
TB View, Threads, Threads with unread The Invisible Man Firefox 1 03-20-2006 02:09 AM
Standard Threads vs Weightless Threads yoda Python 2 08-01-2005 09:12 PM
threads without threads sindica@gmail.com C Programming 4 08-27-2004 09:25 PM



Advertisments