Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > low-end persistence strategies?

Reply
Thread Tools

low-end persistence strategies?

 
 
Paul Rubin
Guest
Posts: n/a
 
      02-16-2005
I've started a few threads before on object persistence in medium to
high end server apps. This one is about low end apps, for example, a
simple cgi on a personal web site that might get a dozen hits a day.
The idea is you just want to keep a few pieces of data around that the
cgi can update.

Immediately, typical strategies like using a MySQL database become too
big a pain. Any kind of compiled and installed 3rd party module (e.g.
Metakit) is also too big a pain. But there still has to be some kind
of concurrency strategy, even if it's something like crude file
locking, or else two people running the cgi simultaneously can wipe
out the data store. But you don't want crashing the app to leave a
lock around if you can help it.

Anyway, something like dbm or shelve coupled with flock-style file
locking and a version of dbmopen that automatically retries after 1
second if the file is locked would do the job nicely, plus there could
be a cleanup mechanism for detecting stale locks.

Is there a standard approach to something like that, or should I just
code it the obvious way?

Thanks.
 
Reply With Quote
 
 
 
 
Diez B. Roggisch
Guest
Posts: n/a
 
      02-16-2005

Maybe ZODB helps.


--
Regards,

Diez B. Roggisch
 
Reply With Quote
 
 
 
 
Thomas Guettler
Guest
Posts: n/a
 
      02-16-2005
Am Tue, 15 Feb 2005 18:57:47 -0800 schrieb Paul Rubin:

> I've started a few threads before on object persistence in medium to
> high end server apps. This one is about low end apps, for example, a
> simple cgi on a personal web site that might get a dozen hits a day.
> The idea is you just want to keep a few pieces of data around that the
> cgi can update.

[cut]
> Anyway, something like dbm or shelve coupled with flock-style file
> locking and a version of dbmopen that automatically retries after 1
> second if the file is locked would do the job nicely, plus there could
> be a cleanup mechanism for detecting stale locks.
>
> Is there a standard approach to something like that, or should I just
> code it the obvious way?


Hi,

I would use the pickle module and access to the
pickle files could be serialized (one after the other
is allowed to read or write) with file locking.

This means your cgi application can only serve one request
after the other.

HTH,
Thomas

--
Thomas GŁttler, http://www.thomas-guettler.de/


 
Reply With Quote
 
Paul Rubin
Guest
Posts: n/a
 
      02-16-2005
"Diez B. Roggisch" <(E-Mail Removed)> writes:
> Maybe ZODB helps.


I think it's way too heavyweight for what I'm envisioning, but I
haven't used it yet. I'm less concerned about object persistence
(just saving strings is good enough) than finding the simplest
possible approach to dealing with concurrent update attempts.
 
Reply With Quote
 
Diez B. Roggisch
Guest
Posts: n/a
 
      02-16-2005
Paul Rubin wrote:

> "Diez B. Roggisch" <(E-Mail Removed)> writes:
>> Maybe ZODB helps.

>
> I think it's way too heavyweight for what I'm envisioning, but I
> haven't used it yet. I'm less concerned about object persistence
> (just saving strings is good enough) than finding the simplest
> possible approach to dealing with concurrent update attempts.


And that's exactly where zodb comes into play. It has full ACID support.
Opening a zodb is a matter of three lines of code - not to be compared to
rdbms'ses. And apart from some standard subclassing, you don't have to do
anything to make your objects persistable. Just check the tutorial.
--
Regards,

Diez B. Roggisch
 
Reply With Quote
 
Paul Rubin
Guest
Posts: n/a
 
      02-16-2005
"Diez B. Roggisch" <(E-Mail Removed)> writes:
> > I think it's way too heavyweight for what I'm envisioning, but I
> > haven't used it yet. I'm less concerned about object persistence
> > (just saving strings is good enough) than finding the simplest
> > possible approach to dealing with concurrent update attempts.

>
> And that's exactly where zodb comes into play. It has full ACID support.
> Opening a zodb is a matter of three lines of code - not to be compared to
> rdbms'ses.


The issue with using an rdbms is not with the small amount of code
needed to connect to it and query it, but in the overhead of
installing the huge piece of software (the rdbms) itself, and keeping
the rdbms server running all the time so the infrequently used app can
connect to it. ZODB is also a big piece of software to install. Is
it at least 100% Python with no C modules required? Does it need a
separate server process? If it needs either C modules or a separate
server, it really can't be called a low-end strategy.
 
Reply With Quote
 
Diez B. Roggisch
Guest
Posts: n/a
 
      02-16-2005
> The issue with using an rdbms is not with the small amount of code
> needed to connect to it and query it, but in the overhead of


Its not only connecting - its creating (automaticall if necessary) and
"connecting" which is actually only opening.

> installing the huge piece of software (the rdbms) itself, and keeping
> the rdbms server running all the time so the infrequently used app can
> connect to it. ZODB is also a big piece of software to install. Is
> it at least 100% Python with no C modules required? Does it need a
> separate server process? If it needs either C modules or a separate
> server, it really can't be called a low-end strategy.


It has to be installed. And it has C-modules - but I don't see that as a
problem. Of course this is my personal opinion - but it's certainly easier
installed than to cough up your own transaction isolated persistence layer.
I started using it over pickle when my multi-threaded app caused pickle to
crash.

ZODB does not have a server-process, and no external setup beyond the
installation of the module itself.

Even if you consider installing it as too heavy for your current needs, you
should skim over the tutorial to get a grasp of how it works.

--
Regards,

Diez B. Roggisch
 
Reply With Quote
 
Tom Willis
Guest
Posts: n/a
 
      02-16-2005
Sounds like you want pickle or cpickle.


On Tue, 15 Feb 2005 19:00:31 -0800 (PST), Paul Rubin
<"http://phr.cx"@nospam.invalid> wrote:
> I've started a few threads before on object persistence in medium to
> high end server apps. This one is about low end apps, for example, a
> simple cgi on a personal web site that might get a dozen hits a day.
> The idea is you just want to keep a few pieces of data around that the
> cgi can update.
>
> Immediately, typical strategies like using a MySQL database become too
> big a pain. Any kind of compiled and installed 3rd party module (e.g.
> Metakit) is also too big a pain. But there still has to be some kind
> of concurrency strategy, even if it's something like crude file
> locking, or else two people running the cgi simultaneously can wipe
> out the data store. But you don't want crashing the app to leave a
> lock around if you can help it.
>
> Anyway, something like dbm or shelve coupled with flock-style file
> locking and a version of dbmopen that automatically retries after 1
> second if the file is locked would do the job nicely, plus there could
> be a cleanup mechanism for detecting stale locks.
>
> Is there a standard approach to something like that, or should I just
> code it the obvious way?
>
> Thanks.
> --
> http://mail.python.org/mailman/listinfo/python-list
>



--
Thomas G. Willis
http://paperbackmusic.net
 
Reply With Quote
 
Paul Rubin
Guest
Posts: n/a
 
      02-16-2005
"Diez B. Roggisch" <(E-Mail Removed)> writes:
> It has to be installed. And it has C-modules - but I don't see that
> as a problem. Of course this is my personal opinion - but it's
> certainly easier installed than to cough up your own transaction
> isolated persistence layer. I started using it over pickle when my
> multi-threaded app caused pickle to crash.


I don't feel that I need ACID since, as mentioned, I'm willing to lock
the entire database for the duration of each transaction. I just want
a simple way to handle locking, retries, and making sure the locks are
cleaned up.

> ZODB does not have a server-process, and no external setup beyond the
> installation of the module itself.


That helps, thanks.

> Even if you consider installing it as too heavy for your current needs, you
> should skim over the tutorial to get a grasp of how it works.


Yes, I've been wanting to look at it sometime.
 
Reply With Quote
 
Paul Rubin
Guest
Posts: n/a
 
      02-16-2005
Tom Willis <(E-Mail Removed)> writes:
> Sounds like you want pickle or cpickle.


No, the issue is how to handle multiple clients trying to update the
pickle simultaneously.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Dataset persistence??? Jason ASP .Net 6 06-07-2010 11:53 AM
Java Persistence API and persistence.xml Kenneth P. Turvey Java 2 03-16-2008 12:08 AM
EJB persistence versus Database persistence? javaguy44 Java 10 05-18-2004 07:08 PM
object persistence Atif Jalal ASP .Net 2 08-12-2003 05:50 PM
Re: Multiple levels of Session Persistence Edwin Kusters ASP .Net 2 07-22-2003 12:12 AM



Advertisments