Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   Growing database? (http://www.velocityreviews.com/forums/t330574-growing-database.html)

Jonathon McKitrick 05-04-2004 12:32 PM

Growing database?
 

This might be a silly question, so I'm warning you ahead of time.

I have a persistence database that is saving a 'month' object out with its
data. A single month pickled is about 50k. My database of shelved objects
contains 4 months of data is over 3 megs. Does the db object journal, or do
some other recordkeeping that could explain the huge size difference between
pickling to a file and pickling to a 'shelve'?

jm
--
My other computer is your Windows box.


David M. Cooke 05-04-2004 05:48 PM

Re: Growing database?
 
At some point, Jonathon McKitrick <jcm@FreeBSD-uk.eu.org> wrote:

> This might be a silly question, so I'm warning you ahead of time.
>
> I have a persistence database that is saving a 'month' object out with its
> data. A single month pickled is about 50k. My database of shelved objects
> contains 4 months of data is over 3 megs. Does the db object journal, or do
> some other recordkeeping that could explain the huge size difference between
> pickling to a file and pickling to a 'shelve'?


Depends on what db module shelve uses for its backend. For instance,
gdbm won't shrink a database file unless you specifically tell it to,
so if you have a lot of deletions it will grow.

Easiest way to fix it is to copy all the objects from your shelve to a
new shelve.

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)physics(dot)mcmaster(dot)ca

j_mckitrick 05-04-2004 09:59 PM

Re: Growing database?
 
> Depends on what db module shelve uses for its backend. For instance,
> gdbm won't shrink a database file unless you specifically tell it to,
> so if you have a lot of deletions it will grow.
>
> Easiest way to fix it is to copy all the objects from your shelve to a
> new shelve.


That makes sense. I wasn't sure how deletions worked, but your idea
is a good one. I'm currently running on FreeBSD, so I am using the
bsddb backend.


All times are GMT. The time now is 10:32 AM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.