Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > having problems with open4 and stuck forked processes

Reply
Thread Tools

having problems with open4 and stuck forked processes

 
 
Tim Uckun
Guest
Posts: n/a
 
      09-22-2010
I am running a batch process which uses the wkhtmltoimage-i386 binary
to make screenshots of urls. Unfortunately this is in beta and it
frequently hangs up and takes up 100% of one of the CPUs on the
machine.

I have the following code to try and detect the hung process and kill
it but it doesn't always work and I was wondering if anybody has a
better idea of how to do this. When I run it by testing simple
commands like sleep it works perfectly. In production with this binary
it doesn't seem to always work.

def Util.shell_with_timeout(cmd, seconds = 3600)
#the default timeout is an hour. That's probably way too long

Timeout::timeout(seconds) {
@pid, @stdin, @stdout, @stderr = Open4.popen4(cmd)
ignored, @status = Process::waitpid2 @pid
if @status.exitstatus != 0
raise "Exit Status not zero"
end
}

@stdout ? @stdout.read.strip : ''
rescue Timeout::Error
Process.detach @pid
Process.kill 'SIGKILL', @pid
raise "Process Timed out"
rescue => e
msg = @stderr ? @stderr.read.strip : ''
msg += e.to_s
raise "Error during execution of command #{cmd}\n #{msg}"
end

 
Reply With Quote
 
 
 
 
Robert Klemme
Guest
Posts: n/a
 
      09-22-2010
On Wed, Sep 22, 2010 at 2:31 PM, Tim Uckun <(E-Mail Removed)> wrote:
> I am running a batch process which uses the wkhtmltoimage-i386 binary
> to make screenshots of urls. =A0Unfortunately this is in beta and it
> frequently hangs up and takes up 100% of one of the CPUs on the
> machine.
>
> I have the following code to try and detect the hung process and kill
> it but it doesn't always work and I was wondering if anybody has a
> better idea of how to do this. =A0When I run it by testing simple
> commands like sleep it works perfectly. In production with this binary
> it doesn't seem to always work.


What do you mean by that? Goes the timeout undetected? Can't you
kill the process? Are there any unexpected error messages /
exceptions?

> def Util.shell_with_timeout(cmd, seconds =3D 3600)
> =A0 =A0#the default timeout is an hour. That's probably way too long
>
> =A0 =A0Timeout::timeout(seconds) {
> =A0 =A0 =A0@pid, @stdin, @stdout, @stderr =3D Open4.popen4(cmd)
> =A0 =A0 =A0ignored, @status =3D Process::waitpid2 @pid
> =A0 =A0 =A0if @status.exitstatus !=3D 0
> =A0 =A0 =A0 =A0raise "Exit Status not zero"
> =A0 =A0 =A0end
> =A0 =A0}
>
> =A0 =A0@stdout ? @stdout.read.strip : ''
> =A0rescue Timeout::Error
> =A0 =A0Process.detach @pid
> =A0 =A0Process.kill 'SIGKILL', @pid
> =A0 =A0raise "Process Timed out"
> =A0rescue =3D> e
> =A0 =A0msg =3D =A0@stderr ? =(E-Mail Removed) : ''
> =A0 =A0msg +=3D =A0 e.to_s
> =A0 =A0raise "Error during execution of command #{cmd}\n #{msg}"
> =A0end


A frequent problem with #popen methods is to not read file descriptors
which can make the client hang (i.e. if it writes more than fits into
a pipe). That could be something to check since you are not reading
any of the streams.

Kind regards

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

 
Reply With Quote
 
 
 
 
Tim Uckun
Guest
Posts: n/a
 
      09-22-2010
> What do you mean by that? =C2=A0Goes the timeout undetected? =C2=A0Can't =
you
> kill the process? =C2=A0Are there any unexpected error messages /
> exceptions?
>


Obviously the timeout is not detected. I am not sure about the
exceptions as it happens when I am not looking but I will ramp up the
logging and see if I can trap anything.

>
> A frequent problem with #popen methods is to not read file descriptors
> which can make the client hang (i.e. if it writes more than fits into
> a pipe). =C2=A0That could be something to check since you are not reading
> any of the streams.


If you have any pointers to documentation about this I would really
appreciate it. I know so little about unix processes and pipes and
such.

 
Reply With Quote
 
Robert Klemme
Guest
Posts: n/a
 
      09-23-2010
On 23.09.2010 01:59, Tim Uckun wrote:
>> What do you mean by that? Goes the timeout undetected? Can't you
>> kill the process? Are there any unexpected error messages /
>> exceptions?

>
> Obviously the timeout is not detected.


I don't find that obvious at all from your initial description.

> I am not sure about the
> exceptions as it happens when I am not looking but I will ramp up the
> logging and see if I can trap anything.


You could start by doing

Thread.abort_on_exception = true

at the beginning of your script.

>> A frequent problem with #popen methods is to not read file descriptors
>> which can make the client hang (i.e. if it writes more than fits into
>> a pipe). That could be something to check since you are not reading
>> any of the streams.

>
> If you have any pointers to documentation about this I would really
> appreciate it. I know so little about unix processes and pipes and
> such.


I don't have anything handy but I guess Google will help.

A pipe is basically what it looks like: it's a piece of pipe with you
write to on one end and read from at the other end. At the read end
there is a valve. If nobody reads the valve stays closed and you can't
fill in more at the write end. If you use blocking IO your process
blocks on the system call and won't be active before you read from the
other end. (This is a bit simplistic because it leaves threads and
interpreter implementation out of the way but this is basically what
happens).

http://en.wikipedia.org/wiki/Pipe_%28Unix%29

Kind regards

robert


--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/
 
Reply With Quote
 
elise huard
Guest
Posts: n/a
 
      09-23-2010
> Thread.abort_on_exception = true
>
> at the beginning of your script.


Euhm (asking this because I honestly don't know) will this work for
Processes ? (he's not using Thread)

Elise

 
Reply With Quote
 
Brian Candler
Guest
Posts: n/a
 
      09-23-2010
elise huard wrote:
> Euhm (asking this because I honestly don't know) will this work for
> Processes ? (he's not using Thread)


Timeout::timeout uses a thread internally - and it raises an exception
asynchronously in the main thread, which makes it unsafe in just about
any application you can think of for it.

It would be safer to use select() on the data coming from the child to
wait for the process to terminate (when you read end-of-file)
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Robert Klemme
Guest
Posts: n/a
 
      09-23-2010
On Thu, Sep 23, 2010 at 9:42 AM, elise huard <(E-Mail Removed)> wrote:
>> Thread.abort_on_exception = true
>>
>> at the beginning of your script.

>
> Euhm (asking this because I honestly don't know) will this work for
> Processes ? (he's not using Thread)


But he uses Timeout which AFAIK uses threads internally for monitoring.

Cheers

robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

 
Reply With Quote
 
Tim Uckun
Guest
Posts: n/a
 
      09-24-2010
> It would be safer to use select() on the data coming from the child to
> wait for the process to terminate (when you read end-of-file)



Do you know of any examples on how to do that? I am willing to rewrite
my code obviously.

 
Reply With Quote
 
Tim Uckun
Guest
Posts: n/a
 
      09-24-2010
> It would be safer to use select() on the data coming from the child to
> wait for the process to terminate (when you read end-of-file)


But what if the process hangs?

Wouldn't I need to use timeout to check for that anyway?

 
Reply With Quote
 
Robert Klemme
Guest
Posts: n/a
 
      09-25-2010
On 25.09.2010 00:25, Tim Uckun wrote:
>> It would be safer to use select() on the data coming from the child to
>> wait for the process to terminate (when you read end-of-file)

>
> But what if the process hangs?
>
> Wouldn't I need to use timeout to check for that anyway?


Select can be called with a timeout which guarantees that the call
returns in time regardless whether there is any data available.

Kind regards

robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: forked processes and testing Terry Reedy Python 0 09-12-2012 08:51 PM
Re: forked processes and testing Kushal Kumaran Python 0 09-12-2012 11:31 AM
forked processes and testing andrea crotti Python 0 09-12-2012 10:20 AM
system waiting for launched process AND forked processes Pierre Morel Ruby 5 02-17-2010 09:23 PM
Kill forked processes kmkz Python 4 02-27-2006 09:36 PM



Advertisments