Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > Threads, Queues and possible memory leak

Reply
Thread Tools

Threads, Queues and possible memory leak

 
 
hemant
Guest
Posts: n/a
 
      06-26-2008
I have a simple thread pool of [default] 20 threads running. They read
a task from a queue and execute it and then go back to waiting on the
queue. Task popped from the queue contains a block and gets executed
within thread pool.

Code is here:

require "thread"

class WorkData
attr_accessor :data,:block,:job_key
def initialize(args,job_key,&block)
@data = args
@job_key = job_key
@block = block
end
end

class ThreadPool
attr_accessor :size,:threads,:work_queue
def initialize(size)
@size = size
@threads = []
@work_queue = Queue.new
puts Time.now
@size.times { add_thread }
puts "all threads started: #{Time.now}"
end
def defer(*args,&block)
job_key = Thread.current[:job_key]
@work_queue << WorkData.new(args,job_key,&block)
end

def add_thread
@threads << Thread.new do
Thread.current[:job_key] = nil
while true
task = @work_queue.pop
Thread.current[:job_key] = task.job_key
block_result = run_task(task)
end
end
end
def run_task task
block_arity = task.block.arity
begin
t_data = task.data
result = nil
if block_arity != 0
result = t_data.is_a?(Array) ? (task.block.call(*t_data)) :
(task.block.call(t_data))
else
result = task.block.call
end
return result
rescue
return nil
end
end
end

class SomeLoop
attr_accessor :thread_pool
def initialize
@thread_pool = ThreadPool.new(20)
end

def start
count = 0
counter = 0
Signal.trap("USR1") { request_generator }
loop do
if count < 600
make_requests()
sleep(0.05)
count += 1
end
sleep(0.005)
if (counter%100 == 0 && length != 0)
puts "Queue Length: #{length}"
puts "Count : #{count}"
elsif(length == 0 && count >= 600)
puts "All tasks has been executed"
end
counter += 1
end
end


def request_generator
puts "lets make request"
600.times {
make_requests()
sleep(0.05)
}
end

def length
thread_pool.work_queue.length
end

def some_job count
sleep 0.8
end

def make_requests
100.times { |i|
thread_pool.defer(i,&method(:some_job))
}
end
end
Thread.abort_on_exception = true
a = SomeLoop.new
a.start

Its being used in a live application and I have managed to extract the
problematic part.

Now the problem is, as tasks get added in the queue, my program starts
using more memory and thats fair. But even after all the tasks are
popped out from the queue the memory stays the same and further
addition of tasks leads to more memory usage and gets eventually
restarted by monitoring app.

You can simulate the scenario by running above program and by sending
"USR1" signal to it. Memory usage just keeps climbing.

I have tried detecting the leak with bleakhouse and can paste the dump
if useful.
I have also tried using 'dike', but couldn't get any meaningful dump (
Ara, may be you can throw some pointers).

Any ideas?

 
Reply With Quote
 
 
 
 
ara.t.howard
Guest
Posts: n/a
 
      06-26-2008

On Jun 26, 2008, at 5:36 AM, hemant wrote:

> I have also tried using 'dike', but couldn't get any meaningful dump (
> Ara, may be you can throw some pointers).


will dike you need at least two dumps and then take deltas. typically
you'd actually look at the deltas of deltas to see long term trends,
which is a manual process. do you mean you couldn't dump or that the
dumps didn't make sense?

cheers.

a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being
better. simply reflect on that.
h.h. the 14th dalai lama




 
Reply With Quote
 
 
 
 
Roger Pack
Guest
Posts: n/a
 
      06-26-2008
> Now the problem is, as tasks get added in the queue, my program starts
> using more memory and thats fair. But even after all the tasks are
> popped out from the queue the memory stays the same and further
> addition of tasks leads to more memory usage and gets eventually
> restarted by monitoring app.


If you think there's a 'real' memory leak then valgrind is your friend.
Also trying ruby from svn might help [latest].
-R
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
hemant
Guest
Posts: n/a
 
      06-26-2008
On 6/26/08, ara.t.howard <(E-Mail Removed)> wrote:
>
> On Jun 26, 2008, at 5:36 AM, hemant wrote:
>
>
> > I have also tried using 'dike', but couldn't get any meaningful dump (
> > Ara, may be you can throw some pointers).
> >

>
> will dike you need at least two dumps and then take deltas. typically
> you'd actually look at the deltas of deltas to see long term trends, which
> is a manual process. do you mean you couldn't dump or that the dumps didn't
> make sense?
>


Well, i didn't get any dump at all. I got 0,1 files in log directory,
but they had nothing in it, except [] ( in one of them and other one
was empty). I will try sometime later today.


> If you think there's a 'real' memory leak then valgrind is your friend.
> Also trying ruby from svn might help [latest].


I have tried running with ruby1.9 trunk and leaks seems to occur
really slow. I mean, with some 150 thousand messages in queue, in
ruby1.8 memory shoots up to 400 MB, but with ruby1.9 it stays around
52MB.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Possible memory leak? Pep C++ 11 07-23-2008 12:39 PM
Possible memory leak? Tuvas Python 29 01-30-2006 08:27 PM
possible memory leak? Roman Mashak C Programming 13 08-12-2005 10:57 AM
Authorization Manager, ASP.NET, possible memory Leak John ASP .Net Security 1 12-15-2004 07:28 PM
Authorization Manager, ASP.NET, possible memory leak John ASP .Net 0 12-15-2004 06:43 PM



Advertisments