Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > High volume DRb server errors

Reply
Thread Tools

High volume DRb server errors

 
 
Bryan Duxbury
Guest
Posts: n/a
 
      10-18-2007
I've run into an interesting problem using DRb as the frontend for
publishing one of our services. We have a single Ruby process publishing
a service via DRb. The service is consumed by 100-300 clients at any
given time.

When the server is first started, things seem to go fine. However, after
a while, it appears that things get out of hand. The number of threads
goes from ~250 (1 per client) to ~400. During this period, the server is
still responsive to some requests, though they are quite slow.

Soon after the number of threads tops 400, it turns a corner and just
goes wild all the way up to 1024 threads. The rate of growth between 400
and 1024 is very fast. Once it reaches 1024 threads, it tops out. This
is strange because the clients are continuing to attempt to make
requests, they just aren't being processed anymore.

If, once the server has died, I kill all the clients, eventually, the
thread count will reduce most of the way down, but stop short of the
bottom. Most often, the thread count seems to reach 64, and won't go any
lower. At this point, the ruby process is using 100% of the cpu, though
what it is doing is anyone's guess. It will not become responsive again
until it is restarted. There are also quite a lot of TCP sockets hanging
out in the CLOSE_WAIT state on the server.

I have attached two sample files I've used to recreate this error in a
vacuum. Run test_server.rb, then do "ruby test_client.rb start 50" to
pretty quickly max out the number of threads on the server.

Has anyone encountered this kind of problem before?

It's also worth noting that DRb has no way at all of protecting itself
from too many connections - it will gladly run into the max allowed open
file handles limit and not handle the error correctly. (I increased the
open file handle limit on my production system because I thought that
was the issue. It appears not to be.) Should DRb be extended to have
more safe server features, or is it just not intended to fill that use?

Could anyone recommend a better, lightweight, robust Ruby RPC library?
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
DRB: can't run subshell in drb server? Ittay Dror Ruby 1 10-21-2008 11:28 AM
DRb connection error with more than 250+ DRb services J. Wook Ruby 16 05-16-2007 11:32 AM
More DRb; SSL & DRB & errors Kirk Haines Ruby 0 07-01-2005 06:29 PM
DRb / dRuby - freezes on DRb::DRbUndumped - any ideas? Miles Keaton Ruby 3 03-30-2005 03:37 PM
High volume websites using Python web server software? Irmen de Jong Python 7 10-27-2004 07:10 PM



Advertisments