«

Hi,

I'm experiencing strange behavior in a Ruby app doing
multithreaded I/O, and I seem to have narrowed it down
to my using a timeout { } block around a condition
variable wait. E.g.

timeout(1.0) { @condition_variable.wait(@mutex) }

The whole program is currently still a bit too large
to post, but now that I'm able to reproduce the problem
pretty reliably, I think I could narrow it down to just
a few lines if need be. But first, I wanted to ask if
it's even *supposed* to be reasonable to use timeout
in this fashion.

If it's supposed to be OK to do a timeout around a
condition variable wait, then, I'll be happy to try
to narrow it down to a few lines of code that reproduce
the problem I'm seeing. (I'm seeing threads not being
awakened when signalled. Note that these are
completely different threads with different condition
variables and mutexes than the one the timeout applies to.)

If doing a timeout isn't OK around a condition variable
wait, then, I'd like to ask what alternatives are
recommended? I would very much like to have a timeout
somehow on a condition variable wait.

Thanks for your help !

Regards,

Bill

Bill Kelly wrote:
> Hi,
>
> I'm experiencing strange behavior in a Ruby app doing
> multithreaded I/O, and I seem to have narrowed it down
> to my using a timeout { } block around a condition
> variable wait. E.g.
>
> timeout(1.0) { @condition_variable.wait(@mutex) }
>
> The whole program is currently still a bit too large
> to post, but now that I'm able to reproduce the problem
> pretty reliably, I think I could narrow it down to just
> a few lines if need be. But first, I wanted to ask if
> it's even *supposed* to be reasonable to use timeout
> in this fashion.
>
> If it's supposed to be OK to do a timeout around a
> condition variable wait, then, I'll be happy to try
> to narrow it down to a few lines of code that reproduce
> the problem I'm seeing. (I'm seeing threads not being
> awakened when signalled. Note that these are
> completely different threads with different condition
> variables and mutexes than the one the timeout applies to.)
>
> If doing a timeout isn't OK around a condition variable
> wait, then, I'd like to ask what alternatives are
> recommended? I would very much like to have a timeout
> somehow on a condition variable wait.

Probably wrong, but... when the timeout happens, a TimeoutError is
raised in the thread that called #timeout. If you don't rescue this
error and do something with it, the thread dies (silently, unless
Thread.abort_on_exception is enabled). This may be why threads that have
called #timeout later become unresponsive. (Or maybe I'm totally wrong
and you are catching the TimeoutError somewhere and the problem is
something else.)

For example, the following code simply prints "main done". If you
uncomment the two lines, you also get "thread done".

require 'timeout'

Thread.new do
begin
timeout(0.2) do
sleep
end
# rescue Exception => e
# p e
end
puts "thread done"
end

sleep 0.4
puts "main done"

HTH.

Hi Joel,

From: "Joel VanderWerf" <>
> Bill Kelly wrote:
> >
> > timeout(1.0) { @condition_variable.wait(@mutex) }
> >
> Probably wrong, but... when the timeout happens, a TimeoutError is
> raised in the thread that called #timeout. If you don't rescue this
> error and do something with it, the thread dies (silently, unless
> Thread.abort_on_exception is enabled). This may be why threads that have
> called #timeout later become unresponsive. (Or maybe I'm totally wrong
> and you are catching the TimeoutError somewhere and the problem is
> something else.)

Right, the actual code doing the timeout looks like:

def timed_wait(timeout_secs)
timedout = false
begin
timeout(timeout_secs) { wait }
rescue Timeout::Error
timedout = true
end
!timedout
end

And, interestingly the above is being called only on the
main thread, and the main thread keeps running. But other
thread(s) which have their own mutexes and condition variables,
start "not being awakened" when signalled, sporradically,
depending on how often the above routine is called on the
main thread.


Regards,

Bill

Hi again,

> I'm experiencing strange behavior in a Ruby app doing
> multithreaded I/O, and I seem to have narrowed it down
> to my using a timeout { } block around a condition
> variable wait. E.g.
>
> timeout(1.0) { @condition_variable.wait(@mutex) }


I have gotten the code down to a couple hundred lines,
and am able to reproduce the problem easily on:

ruby 1.8.0 (2003-08-04) [i386-mswin32]
ruby 1.8.2 (2004-07-29) [i386-mswin32]
ruby 1.8.0 (2003-10-24) [i686-linux]

My apologies in advance if I turn out to be doing something
stupid in my own code.
http://bwk.homeip.net/ftp/buffered-io-test3.rb

If I comment out line 216
100.times { global_signal.timed_wait(0.003) }

and uncomment line 213 instead
100.times { sleep(0.003) }

...I don't experience the problem. But as-is, with line 216
in place, the problem I experience is that the background_write
thread will sporradically not awaken, even after being
signalled repeatedly.

If anyone is able to help confirm that this really is a
bug, or, point out that I'm doing something stupid....
I'd really appreciate it.

Thanks for your help!

Regards,

Bill

> > I'm experiencing strange behavior in a Ruby app doing
> > multithreaded I/O, and I seem to have narrowed it down
> > to my using a timeout { } block around a condition
> > variable wait. E.g.
> >
> > timeout(1.0) { @condition_variable.wait(@mutex) }
>
>
> I have gotten the code down to a couple hundred lines,
> and am able to reproduce the problem easily on:
>
> ruby 1.8.0 (2003-08-04) [i386-mswin32]
> ruby 1.8.2 (2004-07-29) [i386-mswin32]
> ruby 1.8.0 (2003-10-24) [i686-linux]
>
> My apologies in advance if I turn out to be doing something
> stupid in my own code.
> http://bwk.homeip.net/ftp/buffered-io-test3.rb

I have a bit more info. The timeout on ConditionVariable#wait
was causing the current thread to be left in the @waiters
queue in ConditionVariable. I have added a line to ensure
Thread.current is removed from @waiters:

class ConditionVariable
def wait(mutex)
begin
mutex.exclusive_unlock do
@waiters.push(Thread.current)
Thread.stop
end
ensure
mutex.lock
@waiters.delete Thread.current # ADDED THIS LINE
end
end
end

Adding this line SEEMS to fix the primary problem I've
been seeing, but I can't explain why. Because although
my condition variable surrounded by the timeout was
indeed filling up with hundreds of references to
Thread.current in @waiters, the problem I'm seeing
is that a totally different thread with a different
condition variable, simply starts not waking up when
signalled. At least, it doesn't wake up promptly,
consistently. I've tried giving it a high priority,
I know the other threads are mostly sleeping anyway,
whatever, ... I'm still trying to make the program
that reproduces the problem smaller... There's probably
a stupid near-deadlock situation that's my own fault
and I'm just not seeing it.

In any case, I think something like the line I've
added above to ConditionVariable#wait needs to be
there. . . .


Regards,

Bill

> > > I'm experiencing strange behavior in a Ruby app doing
> > > multithreaded I/O, and I seem to have narrowed it down
> > > to my using a timeout { } block around a condition
> > > variable wait. E.g.
> > >
> > > timeout(1.0) { @condition_variable.wait(@mutex) }

For all the avid readers of this thread, (ha ha...
just wanted to follow-up that, based on how timeout itself
is implemented, I've now switched to a timed_wait that
takes a similar approach.

Create an alarm thread, to signal the condition variable
we're waiting on, after the specified time elapses:

def timed_wait(timeout_secs)
begin
alarm_th = Thread.start {
sleep timeout_secs
signal
}
wait
ensure
alarm_th.kill if alarm_th and alarm_th.alive?
end
end


This avoids having to have the Timeout::Error exception
be raised while inside #wait itself. (See prior message
in thread for a patch to #wait to clean up its @waiters.)

Still have not been able to explain the weird behavior
seen with threads not waking up using the timeout approach
without the aforementioned patch to wait...

Now moving on to the next prob. I'm seeing which seems
to be an actual hang in the garbage collector (I'm not
loading any custom extensions or C code.) . . . Will
start new exciting thread for that..


Regards,

Bill