Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > DRb connection error with more than 250+ DRb services

Reply
Thread Tools

DRb connection error with more than 250+ DRb services

 
 
J. Wook
Guest
Posts: n/a
 
      05-11-2007

Simple test codes are here:


class TestServer
include DRb:RbUndumped

def initialize
@obj = Array.new
end

def register(addr)
@obj << addr
end

def sum
s = 0
@obj.each do |t|
v = DRbObject.new_with_uri(t).get_value
if v.nil?
puts s.to_s + " & error"
end
s += v
end
return s
end
end

class TestClient
include DRb:RbUndumped

def initialize(addr, server, value)
DRbObject.new_with_uri(server).register(addr)
@value = value
end
def get_value
@value
end
end


uri = "druby://localhost:"
server_uri = uri + "40000"
server = DRb.start_service(server_uri, TestServer.new)

max_size = 300

(1..max_size).each do |t|
client_uri = uri + (40000 + t).to_s
DRb.start_service(client_uri, TestClient.new(client_uri, server_uri,
t))
end

sum = DRbObject.new_with_uri(server_uri).sum
puts sum


For max_size = 10, sum = 55
For max_size = 100, sum = 5050

...

but

For max_size = 300,
DRb:RbConnError exception raised.

try to make another DRb server after that error in the same process....
#<Errno::EMFILE: Too many open files - socket(2)> raised.


How can I open more than 300 DRb connection?
(I need to make about 1,000 connections .... )

--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
 
 
 
Jung wook Son
Guest
Posts: n/a
 
      05-11-2007
Brian Candler wrote:
> On Fri, May 11, 2007 at 05:25:45PM +0900, J. Wook wrote:
>> #<Errno::EMFILE: Too many open files - socket(2)> raised.

>
> What does ulimit -a show? Particularly look at max open files


dev@seoul$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
max rt priority (-r) unlimited
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
 
 
 
Brian Candler
Guest
Posts: n/a
 
      05-11-2007
On Fri, May 11, 2007 at 05:56:21PM +0900, Jung wook Son wrote:
> Brian Candler wrote:
> > On Fri, May 11, 2007 at 05:25:45PM +0900, J. Wook wrote:
> >> #<Errno::EMFILE: Too many open files - socket(2)> raised.

> >
> > What does ulimit -a show? Particularly look at max open files

>
> dev@seoul$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> max nice (-e) 20
> file size (blocks, -f) unlimited
> pending signals (-i) unlimited
> max locked memory (kbytes, -l) unlimited
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) unlimited
> max rt priority (-r) unlimited
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) unlimited
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited


Hmm. Should be enough, unless you're opening other file-like objects in your
program. But you could try raising/lowering ulimit -n to see if it makes a
difference.

 
Reply With Quote
 
Brian Candler
Guest
Posts: n/a
 
      05-11-2007
On Fri, May 11, 2007 at 06:18:48PM +0900, Jung wook Son wrote:
> I am opening 1024+ file-like objects.
> That's the answer exactly.
>
>
> One more question.
> When I try to change ulimit value, I have some error.
>
>
> dev@seou$: ulimit -n 2048
> -bash: ulimit: open files: cannot modify limit: Operation not permitted
>
> dev@seoul$ sudo ulimit -Sn 2048
> sudo: ulimit: command not found


You have to be root to increase it.

There is some config file which sets the default at login time. I can't
remember offhand what it is - you may need to google.

 
Reply With Quote
 
Brian Candler
Guest
Posts: n/a
 
      05-11-2007
On Fri, May 11, 2007 at 10:46:53AM +0100, Brian Candler wrote:
> > dev@seou$: ulimit -n 2048
> > -bash: ulimit: open files: cannot modify limit: Operation not permitted
> >
> > dev@seoul$ sudo ulimit -Sn 2048
> > sudo: ulimit: command not found

>
> You have to be root to increase it.
>
> There is some config file which sets the default at login time. I can't
> remember offhand what it is - you may need to google.


in /etc/security/limits.conf, something like:

brian hard nofile 2048

Then the user must logout and login again.

 
Reply With Quote
 
Jung wook Son
Guest
Posts: n/a
 
      05-11-2007

Thanks, Brian.

I made it.

--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Marcin Raczkowski
Guest
Posts: n/a
 
      05-13-2007
On Friday 11 May 2007 08:25, J. Wook wrote:
> Simple test codes are here:
>
>
> class TestServer
> include DRb:RbUndumped
>
> def initialize
> @obj = Array.new
> end
>
> def register(addr)
> @obj << addr
> end
>
> def sum
> s = 0
> @obj.each do |t|
> v = DRbObject.new_with_uri(t).get_value
> if v.nil?
> puts s.to_s + " & error"
> end
> s += v
> end
> return s
> end
> end
>
> class TestClient
> include DRb:RbUndumped
>
> def initialize(addr, server, value)
> DRbObject.new_with_uri(server).register(addr)
> @value = value
> end
> def get_value
> @value
> end
> end
>
>
> uri = "druby://localhost:"
> server_uri = uri + "40000"
> server = DRb.start_service(server_uri, TestServer.new)
>
> max_size = 300
>
> (1..max_size).each do |t|
> client_uri = uri + (40000 + t).to_s
> DRb.start_service(client_uri, TestClient.new(client_uri, server_uri,
> t))
> end
>
> sum = DRbObject.new_with_uri(server_uri).sum
> puts sum
>
>
> For max_size = 10, sum = 55
> For max_size = 100, sum = 5050
>
> ...
>
> but
>
> For max_size = 300,
> DRb:RbConnError exception raised.
>
> try to make another DRb server after that error in the same process....
> #<Errno::EMFILE: Too many open files - socket(2)> raised.
>
>
> How can I open more than 300 DRb connection?
> (I need to make about 1,000 connections .... )


question probably everyone want's to ask.why the hell you need 1000
connections? dont' tell me you have 1000 servers with drbservers running
somewhere

--
Marcin Raczkowski
---
Friends teach what you should know
Enemies Teach what you have to know

 
Reply With Quote
 
Michal Suchanek
Guest
Posts: n/a
 
      05-13-2007
On 13/05/07, Marcin Raczkowski <(E-Mail Removed)> wrote:
> On Friday 11 May 2007 08:25, J. Wook wrote:
> > Simple test codes are here:
> >
> >
> > class TestServer
> > include DRb:RbUndumped
> >
> > def initialize
> > @obj = Array.new
> > end
> >
> > def register(addr)
> > @obj << addr
> > end
> >
> > def sum
> > s = 0
> > @obj.each do |t|
> > v = DRbObject.new_with_uri(t).get_value
> > if v.nil?
> > puts s.to_s + " & error"
> > end
> > s += v
> > end
> > return s
> > end
> > end
> >
> > class TestClient
> > include DRb:RbUndumped
> >
> > def initialize(addr, server, value)
> > DRbObject.new_with_uri(server).register(addr)
> > @value = value
> > end
> > def get_value
> > @value
> > end
> > end
> >
> >
> > uri = "druby://localhost:"
> > server_uri = uri + "40000"
> > server = DRb.start_service(server_uri, TestServer.new)
> >
> > max_size = 300
> >
> > (1..max_size).each do |t|
> > client_uri = uri + (40000 + t).to_s
> > DRb.start_service(client_uri, TestClient.new(client_uri, server_uri,
> > t))
> > end
> >
> > sum = DRbObject.new_with_uri(server_uri).sum
> > puts sum
> >
> >
> > For max_size = 10, sum = 55
> > For max_size = 100, sum = 5050
> >
> > ...
> >
> > but
> >
> > For max_size = 300,
> > DRb:RbConnError exception raised.
> >
> > try to make another DRb server after that error in the same process....
> > #<Errno::EMFILE: Too many open files - socket(2)> raised.
> >
> >
> > How can I open more than 300 DRb connection?
> > (I need to make about 1,000 connections .... )

>
> question probably everyone want's to ask.why the hell you need 1000
> connections? dont' tell me you have 1000 servers with drbservers running
> somewhere
>


It's been said that it's some 250 connections. For one, the
application might act as some sort of proxy which would double the
number of sockets.

Apparently it either uses some other files of uses about 4
handles/connection. It looks quite a lot and could be probably lowered
but it does not change the fact that 1000 is a safe default for
non-server applications but can be easily reached by servers.

I wonder why such limits are imposed. It is probably some workaround
for a flaw in UNIX design that introduces possible DoS by exhausting
kernel memory/structures. Maybe it was fixed in some kernels (if
that's even possible) but nobody cared to fix the limit as well.

Thanks

Michal

 
Reply With Quote
 
Brian Candler
Guest
Posts: n/a
 
      05-13-2007
On Mon, May 14, 2007 at 12:33:54AM +0900, John Joyce wrote:
> But aren't most socket-type connections really two file descriptors
> (coming and going)?
> So the 250 would already be 500...?


No - a socket is bi-directional.

 
Reply With Quote
 
John Joyce
Guest
Posts: n/a
 
      05-13-2007

On May 14, 2007, at 5:04 AM, Brian Candler wrote:

> On Mon, May 14, 2007 at 12:33:54AM +0900, John Joyce wrote:
>> But aren't most socket-type connections really two file descriptors
>> (coming and going)?
>> So the 250 would already be 500...?

>
> No - a socket is bi-directional.
>

Ok. I had that mixed up then with some other networking thing...
(good thing I don't do sockets. or, maybe I should play with it some
soon)

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Like all great travelers, I have seen more than I remember andremember more than I have seen. shenrilaa@gmail.com Java 0 03-06-2008 08:11 AM
Like all great travelers, I have seen more than I remember andremember more than I have seen. shenrilaa@gmail.com C++ 0 03-05-2008 08:41 AM
Like all great travelers, I have seen more than I remember andremember more than I have seen. shenrilaa@gmail.com C Programming 0 03-05-2008 03:26 AM
More DRb; SSL & DRB & errors Kirk Haines Ruby 0 07-01-2005 06:29 PM
DRb / dRuby - freezes on DRb::DRbUndumped - any ideas? Miles Keaton Ruby 3 03-30-2005 03:37 PM



Advertisments