Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > How to pick up dropped script automatically, active records.

Reply
Thread Tools

How to pick up dropped script automatically, active records.

 
 
Marc Hoeppner
Guest
Posts: n/a
 
      09-10-2008
Hi!

I have been using Ruby for a while now - so I have a reasonable
understanding of how it works.
But I have a problem that I can't seem to be able to solve on my own, so
here goes:

I am using an active records based script that queries a remote MySQL
database and retrieves data. It's quite an extensive search, meaning it
takes a long time to run. Unfortunately, the connection closes at random
(I suppose there is some sort of time-out if a connection has been
existing for a certain amount of time).
I have the means of identifying the current ID of the last object and
pick up the search where it left off - manually. And that's the problem:
How to write a wrapper or some loop that:
a) Detects if the database connections closes
b) keeps track of the id of the last processed database object (I
suppose that can be stored in a variable, if we are talking about a
loop, otherwise a "return id"like command (?)
c) re-initializes the script using the last id

I am thinking either a second script, that starts the original
database-search-script and can keep track of its current status or else
a while-loop perhaps?
Since the script simply aborts when the connection closes, I suppose the
first solution would be better?
But I have no idea how to:

- keep track of the "health" of the original process (there is obviously
an error message returned to STDOUT, but I am not sure how to pick it
up)
- start a ruby script from within a ruby script
- or else how to write a loop that keeps track of the child process
without the whole process being terminated by an error

Any help would be greatly appreciated!

Cheers,

Marc
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
 
 
 
Mikel Lindsaar
Guest
Posts: n/a
 
      09-10-2008
On Wed, Sep 10, 2008 at 2:50 PM, Marc Hoeppner
<(E-Mail Removed)> wrote:
> I am using an active records based script that queries a remote MySQL
> database and retrieves data. It's quite an extensive search, meaning it
> takes a long time to run. Unfortunately, the connection closes at random
> (I suppose there is some sort of time-out if a connection has been
> existing for a certain amount of time).


I recently made a similar system.

The best way i found to do it is to make a table at one end that holds
all the rows "to be replicated" or in your case "has been replicated".
This holds the ID of the field you have replicated

Then you write your script to grab the id to replicate, replicate it
down, and if all was successful, put an entry in your 'has been
replicated' table with the ID. Then the next time your script runs it
checks what the last ID was that was successfully replicated and picks
up from there.

Put your replication code at the local end in a transaction so that
you only write into the has been replicated table IF no errors came
up, and you don't replicate anything unless the whole thing completes
successfully.

This then survives restarting hung processes etc.

On detecting health, I found the best way is to set a timeout on each
replication check, and then nuke the process if it takes too long. If
you have used the above pattern on your replication, this won't hurt
anything.

Warning though, Ruby's timeout will not handle this as the DB call is
a system call and will not be killed by Ruby timeout. See my previous
thread in Ruby-talk "Frustrated about system timeouts". The upshot is
that Ara and I are wrapping up a new library called Terminator that
will handle the timeout correctly. But you could adapt the code in
that thread to your problem right now.

Hope that helps.

Mikel

--
http://lindsaar.net/
Rails, RSpec and Life blog....

 
Reply With Quote
 
 
 
 
Mark Thomas
Guest
Posts: n/a
 
      09-10-2008
What you want is a message queue solution. These do exactly what you
want. Your main script can dole out tasks, and workers pick the tasks
up, complete them, and report in. An added bonus is that this can give
you some parallelism, potentially speeding up your search.

Take a look at Starling:
http://rubypond.com/articles/2008/07...g-up-starling/

Drb:
http://chadfowler.com/ruby/drb.html
http://segment7.net/projects/ruby/drb/introduction.html

AP4R:
http://ap4r.rubyforge.org/wiki/wiki.pl?HomePage

in addition to Rinda, bj, beanstalkd, sparrow, rq...

-- Mark.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
Python as active script <script language=python> misterdi Python 0 07-28-2009 04:30 PM
When to pick quad core and when to pick dual core thingy NZ Computing 6 11-21-2006 07:08 AM
When to pick ASP.Net, when to pick desktop? tom c ASP .Net 5 11-01-2006 06:15 PM
script.src pick out Oliver Albrecht Javascript 2 04-27-2006 12:54 PM



Advertisments