Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Managing a queue of subprocesses?

Reply
Thread Tools

Managing a queue of subprocesses?

 
 
cypher543
Guest
Posts: n/a
 
      12-30-2006
My app uses a "queue" of commands which are run one at a time. I am
using the subprocess module to execute the commands in the queue.
However, processes always run at the same time. How can I make one
process run at a time, and then execute the next process when the first
has terminated? My code is below:

self.cmdQueue = {}
self.queue(theProject.directory, "ls", "-l")
self.queue(theProject.directory, "echo", "hello, world!")

def consoleLogAddLine(self, text):
self.consoleLogBuffer.insert(self.consoleLogBuffer .get_end_iter(),
text)
self.consoleLog.scroll_to_mark(self.consoleLogBuff er.get_insert(), 0)

def onGetData(self, fd, cond, *args):
self.consoleLogAddLine(fd.readline())
return True

def queue(self, rootDir, cmd, args = ""):
count = len(self.cmdQueue) + 1
self.cmdQueue[count] = [cmd, args, rootDir]

def runQueue(self):
for i in self.cmdQueue.values():
self.execute(i[2], i[0], i[1])

def execute(self, rootDir, cmd, args = ""):
os.chdir(rootDir)
if args == "":
buildCmd = cmd
else:
args = args.split(" ")
buildCmd = [cmd] + args
self.buildPID = subprocess.Popen(buildCmd, stdout = subprocess.PIPE,
stderr = subprocess.STDOUT)
gobject.io_add_watch(self.buildPID.stdout, gobject.IO_IN,
self.onGetData)

As you can see, I add the commands "ls -l" and "echo Hello" to the
queue. However, "Hello" is always printed inside the output of "ls -l".
I would like to wait for "ls -l" to terminate and then run "echo
Hello". But, the output must still print to the consoleLogBuffer
line-by-line, and my GUI must not hang during execution.

Is this even possible?

 
Reply With Quote
 
 
 
 
Tom Plunket
Guest
Posts: n/a
 
      12-30-2006
cypher543 wrote:

> self.buildPID = subprocess.Popen(buildCmd, stdout = subprocess.PIPE, stderr = subprocess.STDOUT)


Instead of calling it self.buildPID, you might just call it
self.buildProcess or something. It's actually a Popen object that gets
returned.

So yes you can do what you want:

__init__ self.buildProcess to None. Then, in execute(), if
(self.buildProcess is None) or (self.buildProcess.poll() is not None)
start the next process.

You've got to wait for one process to end before starting the next one,
it's really that easy. So, don't just go ahead and fire them all
instantly. Possibly what you want to do instead is have runQueue() do
the check somehow that there's no active process running.

What I would do, have runQueue() check to see if self.buildProcess is
None. If it is None, fire the next process in the queue. If it isn't
None, then check to see if it's ended. If it has ended, then set
self.buildProcess to None. Next UI update the next step in the queue
gets done. Mind that you'll have to modify the queue as you go, e.g.
self.queue = self.queue[1:].

Finally, consider piping stderr separately, and direct its output to a
different window in your GUI. You could even make that window pop open
on demand, if errors occur.

good luck,
-tom!

--
 
Reply With Quote
 
 
 
 
cypher543
Guest
Posts: n/a
 
      12-30-2006
That was a very good answer, and it sure sounds like it would work.
However, I failed at implementing it. My updated runQueue() function
is:

def runQueue(self):
self.buildProcess = None
count = 1 # current position in the queue
while True:
if self.buildProcess is None:
self.execute(self.cmdQueue[count][2], self.cmdQueue[count][0],
self.cmdQueue[count][1])
count = count + 1
else:
# I'm not really sure what to put here

I pretty sure I did all of that wrong. Also, how exactly would I
redirect stderr to another place?

On Dec 30, 12:22 am, Tom Plunket <(E-Mail Removed)> wrote:
> cypher543 wrote:
> > self.buildPID = subprocess.Popen(buildCmd, stdout = subprocess.PIPE, stderr = subprocess.STDOUT)Instead of calling it self.buildPID, you might just call it

> self.buildProcess or something. It's actually a Popen object that gets
> returned.
>
> So yes you can do what you want:
>
> __init__ self.buildProcess to None. Then, in execute(), if
> (self.buildProcess is None) or (self.buildProcess.poll() is not None)
> start the next process.
>
> You've got to wait for one process to end before starting the next one,
> it's really that easy. So, don't just go ahead and fire them all
> instantly. Possibly what you want to do instead is have runQueue() do
> the check somehow that there's no active process running.
>
> What I would do, have runQueue() check to see if self.buildProcess is
> None. If it is None, fire the next process in the queue. If it isn't
> None, then check to see if it's ended. If it has ended, then set
> self.buildProcess to None. Next UI update the next step in the queue
> gets done. Mind that you'll have to modify the queue as you go, e.g.
> self.queue = self.queue[1:].
>
> Finally, consider piping stderr separately, and direct its output to a
> different window in your GUI. You could even make that window pop open
> on demand, if errors occur.
>
> good luck,
> -tom!
>
> --


 
Reply With Quote
 
Tom Plunket
Guest
Posts: n/a
 
      01-01-2007
cypher543 wrote:

> That was a very good answer, and it sure sounds like it would work.
> However, I failed at implementing it. My updated runQueue() function
> is:
>
> def runQueue(self):
> self.buildProcess = None
> count = 1 # current position in the queue
> while True:
> if self.buildProcess is None:
> self.execute(self.cmdQueue[count][2], self.cmdQueue[count][0],
> self.cmdQueue[count][1])
> count = count + 1
> else:
> # I'm not really sure what to put here
>
> I pretty sure I did all of that wrong. Also, how exactly would I
> redirect stderr to another place?


You're thinking too hard.

class Whatever:
def __init__(self):
self.process = None
# and other stuff, probably.

def runQueue(self):
# check to see if no process has been started, or if
# the most-recently-started one has finished.
if not (self.process or (self.process.poll() is None)):
if self.process:
previousReturnCode = self.process.returncode

if len(self.cmdQueue) > 0:
command = self.cmdQueue.pop(0) # pull off the first command.
self.execute(command[2], command[0], command[1])
else:
self.process = None

....then, to prevent your GUI from freezing, you need to just call this
function in your idle handling. If you don't want to start all of your
commands at once, you need to not start all your commands at once.


-tom!

--
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Program blocked in Queue.Queue.get and Queue.Queue.put Kris Python 0 01-04-2012 03:46 PM
why does the following with Queue, q.put('\x02', True) not put itin the queue? Gabriel Rossetti Python 3 04-25-2008 03:41 PM
Is Queue.Queue.queue.clear() thread-safe? Russell Warren Python 4 06-27-2006 03:03 PM
what's the difference between #include "queue.h" and #include "queue.cpp" Kceiw C++ 3 03-14-2006 03:01 AM
Queue.Queue-like class without the busy-wait Paul L. Du Bois Python 29 04-04-2005 01:28 PM



Advertisments