Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > Looking for a Fast Persistent Store

Reply
Thread Tools

Looking for a Fast Persistent Store

 
 
Bob Hutchison
Guest
Posts: n/a
 
      08-09-2006
Hi,

I'm looking for a persistent store, where:
* I can use it from Ruby
* it is fast
* it is transactional
* it can update more than one store in a single transaction
* simple String -> String mappings in the store are sufficient
* ideally uses files in the filesystem

These kinds of stores are normally implemented as persistent hashes
or BTrees (or both). I know that Sleepycat's Berkeley DB <http://
www.sleepycat.com/> does this, and I've used Java based systems that
do this. I also know of some C based things but they don't have Ruby
wrappers. I can't find anything in the Ruby world, and I don't care
too much if it isn't pure Ruby.

I know about Purple <http://purple.rubyforge.org/> and the QDMB Ruby
wrapper <http://qdbm.sourceforge.net/>. Neither do the multiple hash/
BTree in a transaction thing.

The trick appears to be with transactions.

I know this can be done easily enough in a Relational DB, but I know
that, for example, JDBC/MySQL combination is significantly slower at
what I need to do than Perst (a pure Java thing that's startlingly
fast). I've taken a look at sqlite, which satisfies all requirements
but I can't shake this feeling that things can be better.

Does anyone have any ideas?

Thanks,
Bob

----
Bob Hutchison -- blogs at <http://www.recursive.ca/
hutch/>
Recursive Design Inc. -- <http://www.recursive.ca/>
Raconteur -- <http://www.raconteur.info/>
xampl for Ruby -- <http://rubyforge.org/projects/xampl/>




 
Reply With Quote
 
 
 
 
Harold Hausman
Guest
Posts: n/a
 
      08-09-2006
On 8/9/06, Bob Hutchison <(E-Mail Removed)> wrote:
> Hi,
>
> I'm looking for a persistent store, where:
> * I can use it from Ruby
> * it is fast
> * it is transactional
> * it can update more than one store in a single transaction
> * simple String -> String mappings in the store are sufficient
> * ideally uses files in the filesystem
>

...
>
> Does anyone have any ideas?
>
> Thanks,
> Bob
>


Hi Bob,

If you're the fun and adventurous type you might want to look into
Mongoose. ( http://rubyforge.org/projects/mongoose/ ) It's new, and
claims to be fast. In the past I wrote some small apps that used the
same authors other system called KirbyBase which was very nice, I
haven't tried Mongoose yet, but I'm happily awaiting a problem with
which to use it on. Though now that I'm thinking about it I don't
know if it has built-in "transaction" support. I suppose it depends on
what you want it to do.

Have you tried SQLite and found it to be too slow? It sounds to me
like you might be prematurely optimizing. We have several medium to
medium-large sized apps that make heavy use of SQLite and speed has
been not a problem at all.

What domain are you working in that requires this kind of speed
anyway? Just curious.

Hope that helps,
-Harold

 
Reply With Quote
 
 
 
 
Robert Klemme
Guest
Posts: n/a
 
      08-09-2006
On 09.08.2006 15:30, Bob Hutchison wrote:
> Hi,
>
> I'm looking for a persistent store, where:
> * I can use it from Ruby
> * it is fast
> * it is transactional
> * it can update more than one store in a single transaction
> * simple String -> String mappings in the store are sufficient
> * ideally uses files in the filesystem


What about PStore?

robert
 
Reply With Quote
 
ara.t.howard@noaa.gov
Guest
Posts: n/a
 
      08-09-2006
On Wed, 9 Aug 2006, Bob Hutchison wrote:

> Hi,
>
> I'm looking for a persistent store, where:
> * I can use it from Ruby
> * it is fast
> * it is transactional
> * it can update more than one store in a single transaction
> * simple String -> String mappings in the store are sufficient
> * ideally uses files in the filesystem
>
> These kinds of stores are normally implemented as persistent hashes or
> BTrees (or both). I know that Sleepycat's Berkeley DB
> <http://www.sleepycat.com/> does this, and I've used Java based systems that
> do this. I also know of some C based things but they don't have Ruby
> wrappers. I can't find anything in the Ruby world, and I don't care too much
> if it isn't pure Ruby.
>
> I know about Purple <http://purple.rubyforge.org/> and the QDMB Ruby wrapper
> <http://qdbm.sourceforge.net/>. Neither do the multiple hash/BTree in a
> transaction thing.
>
> The trick appears to be with transactions.


check out joel's fsdb - it's very nice if you want to go pure ruby. i've used
it on several projects.

> I know this can be done easily enough in a Relational DB, but I know that,
> for example, JDBC/MySQL combination is significantly slower at what I need
> to do than Perst (a pure Java thing that's startlingly fast). I've taken a
> look at sqlite, which satisfies all requirements but I can't shake this
> feeling that things can be better.
>
> Does anyone have any ideas?


sqlite is hard to beat - i use it in at least 20 production systems and have
never had a single error. ruby queue uses it under the hood for the cluster
job store and this is used over nfs - we've been through disk failures and
power failures and come through unscathed.

it's also very fast, esp if you use in memory tables, but on linux it's just
as easy to copy the db to /dev/shm and then go...

if you end up using it try with my arrayfields package - james' sqlite binding
detects it automatically and the tuples with come back as arrays with named
field access - it's faster and requires less memory than a hash, and is also
really convenient to code with.

cheers.

-a
--
to foster inner awareness, introspection, and reasoning is more efficient than
meditation and prayer.
- h.h. the 14th dali lama

 
Reply With Quote
 
M. Edward (Ed) Borasky
Guest
Posts: n/a
 
      08-09-2006
Harold Hausman wrote:
> Have you tried SQLite and found it to be too slow? It sounds to me
> like you might be prematurely optimizing. We have several medium to
> medium-large sized apps that make heavy use of SQLite and speed has
> been not a problem at all.

On Linux, at any rate, one ought to be able to make SQLite extremely
fast by adding RAM and tuning the I/O subsystem and the kernel, and
using tricks like memory mapped files, assuming the database in question
isn't too large. I'm assuming it's small, otherwise he'd pretty much
need a humongous database server and an industrial strength database
like Oracle or PostgreSQL.

 
Reply With Quote
 
Bob Hutchison
Guest
Posts: n/a
 
      08-09-2006

On Aug 9, 2006, at 9:56 AM, Harold Hausman wrote:
> Hi Bob,
>
> If you're the fun and adventurous type you might want to look into
> Mongoose. ( http://rubyforge.org/projects/mongoose/ ) It's new, and
> claims to be fast. In the past I wrote some small apps that used the
> same authors other system called KirbyBase which was very nice, I
> haven't tried Mongoose yet, but I'm happily awaiting a problem with
> which to use it on. Though now that I'm thinking about it I don't
> know if it has built-in "transaction" support. I suppose it depends on
> what you want it to do.


I have had a look at Mongoose, and quite like it. I'm sure to find a
use for it. However, for this particular situation Mongoose's
transactions are too week.

>
> Have you tried SQLite and found it to be too slow? It sounds to me
> like you might be prematurely optimizing. We have several medium to
> medium-large sized apps that make heavy use of SQLite and speed has
> been not a problem at all.


In the Java world things like Perst and JDBM were very fast. SQLite
was problematic for some reason that is a bit foggy just now (I'll
think of it).

>
> What domain are you working in that requires this kind of speed
> anyway? Just curious.


I write web based applications. The situations I'm thinking of are
either financial or CMS-like applications. In the financial cases
I've got situations where there might be a few million (probably not
more than ten million, well maybe 20 million) small things tucked
away. In the case of CMS-like stuff there are far fewer things, lets
say 25k at most and usually more like a thousand or so, but they are
bigger things.

The key factor (pardon the pun) is that the *vast* majority of the
time (and I know how to make it 100% of the time) queries are as
you'd have in a hash table (i.e. a single key where the key is
usually a String).

This is for the persistent store associated with xampl (see URL in my
signature). Xampl already has a few persistent stores but I now need
one a little fancier.


>
> Hope that helps,
> -Harold
>


Thanks Harold.

Cheers,
Bob

----
Bob Hutchison -- blogs at <http://www.recursive.ca/
hutch/>
Recursive Design Inc. -- <http://www.recursive.ca/>
Raconteur -- <http://www.raconteur.info/>
xampl for Ruby -- <http://rubyforge.org/projects/xampl/>




 
Reply With Quote
 
Bob Hutchison
Guest
Posts: n/a
 
      08-09-2006

On Aug 9, 2006, at 10:05 AM, Robert Klemme wrote:

> On 09.08.2006 15:30, Bob Hutchison wrote:
>> Hi,
>> I'm looking for a persistent store, where:
>> * I can use it from Ruby
>> * it is fast
>> * it is transactional
>> * it can update more than one store in a single transaction
>> * simple String -> String mappings in the store are sufficient
>> * ideally uses files in the filesystem

>
> What about PStore?


Oh. Right. PStore never registered with me before.

I just had a look at it and it seems to be along the lines of QDBM
and Purple, transactional on a single store and each store being one
hash/tree.

Thanks for pointing that out.

Cheers,
Bob

>
> robert
>


----
Bob Hutchison -- blogs at <http://www.recursive.ca/
hutch/>
Recursive Design Inc. -- <http://www.recursive.ca/>
Raconteur -- <http://www.raconteur.info/>
xampl for Ruby -- <http://rubyforge.org/projects/xampl/>




 
Reply With Quote
 
Bob Hutchison
Guest
Posts: n/a
 
      08-09-2006
Hi Ara,

On Aug 9, 2006, at 10:08 AM, http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:

> On Wed, 9 Aug 2006, Bob Hutchison wrote:
>
>> Hi,
>>
>> I'm looking for a persistent store, where:
>> * I can use it from Ruby
>> * it is fast
>> * it is transactional
>> * it can update more than one store in a single transaction
>> * simple String -> String mappings in the store are sufficient
>> * ideally uses files in the filesystem
>>
>> These kinds of stores are normally implemented as persistent
>> hashes or
>> BTrees (or both). I know that Sleepycat's Berkeley DB
>> <http://www.sleepycat.com/> does this, and I've used Java based
>> systems that
>> do this. I also know of some C based things but they don't have Ruby
>> wrappers. I can't find anything in the Ruby world, and I don't
>> care too much
>> if it isn't pure Ruby.
>>
>> I know about Purple <http://purple.rubyforge.org/> and the QDMB
>> Ruby wrapper <http://qdbm.sourceforge.net/>. Neither do the
>> multiple hash/BTree in a transaction thing.
>>
>> The trick appears to be with transactions.

>
> check out joel's fsdb - it's very nice if you want to go pure
> ruby. i've used
> it on several projects.


I'm already using that (version 0.5 -- I can't get to RAA right now
for some reason and there are no files on Rubyforge for fsdb so I
don't know if there is a more recent version). In version 0.5 the
transactions were not sufficient I think (it would be nice if I was
wrong).

>
>> I know this can be done easily enough in a Relational DB, but I
>> know that,
>> for example, JDBC/MySQL combination is significantly slower at
>> what I need
>> to do than Perst (a pure Java thing that's startlingly fast). I've
>> taken a
>> look at sqlite, which satisfies all requirements but I can't shake
>> this
>> feeling that things can be better.
>>
>> Does anyone have any ideas?

>
> sqlite is hard to beat - i use it in at least 20 production systems
> and have
> never had a single error. ruby queue uses it under the hood for
> the cluster
> job store and this is used over nfs - we've been through disk
> failures and
> power failures and come through unscathed.


That's good to hear. Don't misunderstand, I *like* SQLite but I don't
know that it is suitable. If I can't find anything else I'll use it.

The thing that makes me nervous is that I can do what I need with
just two tables. One that has (key, value-kind, value), and another
that has (index-name, index-value, key, value-kind). Each column is a
String. The vast majority of queries would be based on key and value-
kind on the first table. The remaining queries would be a select/join
kind of thing. And I can very easily jam the key and value-kind into
the same field (but that would be an optimisation that may not be
necessary). The trouble is that the first table might get to be very
very large. I don't know how SQLite behaves with a single huge table.
I suppose I'm going to have to find out.

>
> it's also very fast, esp if you use in memory tables, but on linux
> it's just
> as easy to copy the db to /dev/shm and then go...


That's interesting.

>
> if you end up using it try with my arrayfields package - james'
> sqlite binding
> detects it automatically and the tuples with come back as arrays
> with named
> field access - it's faster and requires less memory than a hash,
> and is also
> really convenient to code with.
>


I noticed that the other day. Nice.

Cheers,
Bob

> cheers.
>
> -a
> --
> to foster inner awareness, introspection, and reasoning is more
> efficient than
> meditation and prayer.
> - h.h. the 14th dali lama
>


----
Bob Hutchison -- blogs at <http://www.recursive.ca/
hutch/>
Recursive Design Inc. -- <http://www.recursive.ca/>
Raconteur -- <http://www.raconteur.info/>
xampl for Ruby -- <http://rubyforge.org/projects/xampl/>




 
Reply With Quote
 
Bob Hutchison
Guest
Posts: n/a
 
      08-09-2006

On Aug 9, 2006, at 10:15 AM, M. Edward (Ed) Borasky wrote:

> Harold Hausman wrote:
>> Have you tried SQLite and found it to be too slow? It sounds to me
>> like you might be prematurely optimizing. We have several medium to
>> medium-large sized apps that make heavy use of SQLite and speed has
>> been not a problem at all.

> On Linux, at any rate, one ought to be able to make SQLite
> extremely fast by adding RAM and tuning the I/O subsystem and the
> kernel, and using tricks like memory mapped files, assuming the
> database in question isn't too large. I'm assuming it's small,
> otherwise he'd pretty much need a humongous database server and an
> industrial strength database like Oracle or PostgreSQL.
>


A few GB max for one set of applications, far far less for others.

Cheers,
Bob

----
Bob Hutchison -- blogs at <http://www.recursive.ca/
hutch/>
Recursive Design Inc. -- <http://www.recursive.ca/>
Raconteur -- <http://www.raconteur.info/>
xampl for Ruby -- <http://rubyforge.org/projects/xampl/>




 
Reply With Quote
 
ara.t.howard@noaa.gov
Guest
Posts: n/a
 
      08-09-2006
On Thu, 10 Aug 2006, Bob Hutchison wrote:

> The thing that makes me nervous is that I can do what I need with just two
> tables. One that has (key, value-kind, value), and another that has
> (index-name, index-value, key, value-kind). Each column is a String. The
> vast majority of queries would be based on key and value-kind on the first
> table. The remaining queries would be a select/join kind of thing. And I
> can very easily jam the key and value-kind into the same field (but that
> would be an optimisation that may not be necessary). The trouble is that the
> first table might get to be very very large. I don't know how SQLite behaves
> with a single huge table. I suppose I'm going to have to find out.


please post a summary of your findings when you run your tests - i'd be
interested to read them.

cheers.

-a
--
to foster inner awareness, introspection, and reasoning is more efficient than
meditation and prayer.
- h.h. the 14th dali lama

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Persistent field and Persistent properties - difference gk Java 7 10-12-2010 09:43 PM
A pro dofus kamas store,cheap and fast deliver i love runescape Computer Support 0 07-18-2007 06:17 AM
How to store/load persistent data? Josef Wolf Ruby 0 09-04-2006 05:30 PM
to store or not to store an image =?Utf-8?B?UnVkeQ==?= ASP .Net 6 03-30-2005 05:51 AM
Cache to store persistent application's variables NWx ASP .Net 2 02-19-2004 08:07 PM



Advertisments