Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > shelf-like list?

Reply
Thread Tools

shelf-like list?

 
 
kj
Guest
Posts: n/a
 
      08-10-2010



I'm looking for a module that implements "persistent lists": objects
that behave like lists except that all their elements are stored
on disk. IOW, the equivalent of "shelves", but for lists rather
than a dictionaries.

Does anyone know of such a module?

(I suppose that I could slap together a crude implementation of
such a thing by wrapping a shelf with suitable methods to simulate
the list interface. But I'd rather not roll my own if a tested
implementation already exist.)

TIA!

~K
 
Reply With Quote
 
 
 
 
Thomas Jollans
Guest
Posts: n/a
 
      08-12-2010
On Tuesday 10 August 2010, it occurred to kj to exclaim:
> I'm looking for a module that implements "persistent lists": objects
> that behave like lists except that all their elements are stored
> on disk. IOW, the equivalent of "shelves", but for lists rather
> than a dictionaries.
>
> Does anyone know of such a module?
>
> (I suppose that I could slap together a crude implementation of
> such a thing by wrapping a shelf with suitable methods to simulate
> the list interface. But I'd rather not roll my own if a tested
> implementation already exist.)


You could simply use pickle to save the data every once in a while.
 
Reply With Quote
 
 
 
 
Raymond Hettinger
Guest
Posts: n/a
 
      08-13-2010
On Aug 12, 1:37*pm, Thomas Jollans <(E-Mail Removed)> wrote:
> On Tuesday 10 August 2010, it occurred to kj to exclaim:
>
> > I'm looking for a module that implements "persistent lists": objects
> > that behave like lists except that all their elements are stored
> > on disk. *IOW, the equivalent of "shelves", but for lists rather
> > than a dictionaries.

. . .
> You could simply use pickle to save the data every once in a while.


That is a very reasonable solution.



 
Reply With Quote
 
kj
Guest
Posts: n/a
 
      08-15-2010
In <(E-Mail Removed)> Raymond Hettinger <(E-Mail Removed)> writes:

>On Aug 12, 1:37=A0pm, Thomas Jollans <(E-Mail Removed)> wrote:
>> On Tuesday 10 August 2010, it occurred to kj to exclaim:
>>
>> > I'm looking for a module that implements "persistent lists": objects
>> > that behave like lists except that all their elements are stored
>> > on disk. =A0IOW, the equivalent of "shelves", but for lists rather
>> > than a dictionaries.

> . . .
>> You could simply use pickle to save the data every once in a while.


>That is a very reasonable solution.


Sorry I don't follow. Some sample code would be helpful.
TIA,
~K

 
Reply With Quote
 
Chris Rebert
Guest
Posts: n/a
 
      08-15-2010
On Sat, Aug 14, 2010 at 5:13 PM, kj <(E-Mail Removed)> wrote:
> In <(E-Mail Removed)> Raymond Hettinger <(E-Mail Removed)> writes:
>>On Aug 12, 1:37=A0pm, Thomas Jollans <(E-Mail Removed)> wrote:
>>> On Tuesday 10 August 2010, it occurred to kj to exclaim:
>>>
>>> > I'm looking for a module that implements "persistent lists": objects
>>> > that behave like lists except that all their elements are stored
>>> > on disk. =A0IOW, the equivalent of "shelves", but for lists rather
>>> > than a dictionaries.

>> . . .
>>> You could simply use pickle to save the data every once in a while.

>
>>That is a very reasonable solution.

>
> Sorry I don't follow. *Some sample code would be helpful.


I would assume something along the lines of (untested):

from pickle import dump

MOD_THRESHOLD = 42

class PersistentList(list):
def __init__(self, filepath):
self.filepath = filepath
self._mod_count = 0

def save(self):
with open(self.filepath, 'w') as f:
dump(self, f)
self._mod_count = 0

def append(self, *args, **kwds):
super(PersistentList, self).append(*args, **kwds)
self._mod_count += 1
if self._mod_count >= MOD_THRESHOLD:
# obviously time-since-last-dump or other
# more sophisticated metrics might be used instead
self.save()
# define similar wrappers for list's other mutator methods:
__delitem__, __iadd__, __imul__, __setitem__, extend, insert, pop,
remove, etc.
# using decorators should help decrease code duplication

Cheers,
Chris
--
http://blog.rebertia.com
 
Reply With Quote
 
Martin v. Loewis
Guest
Posts: n/a
 
      08-15-2010
> Does anyone know of such a module?

ZODB supports persistent lists.

Regards,
Martin
 
Reply With Quote
 
kj
Guest
Posts: n/a
 
      08-15-2010
In <(E-Mail Removed)> Chris Rebert <(E-Mail Removed)> writes:

>On Sat, Aug 14, 2010 at 5:13 PM, kj <(E-Mail Removed)> wrote:
>> In <(E-Mail Removed)> Ra=

>ymond Hettinger <(E-Mail Removed)> writes:
>>>On Aug 12, 1:37=3DA0pm, Thomas Jollans <(E-Mail Removed)> wrote:
>>>> On Tuesday 10 August 2010, it occurred to kj to exclaim:
>>>>
>>>> > I'm looking for a module that implements "persistent lists": objects
>>>> > that behave like lists except that all their elements are stored
>>>> > on disk. =3DA0IOW, the equivalent of "shelves", but for lists rather
>>>> > than a dictionaries.
>>> . . .
>>>> You could simply use pickle to save the data every once in a while.

>>
>>>That is a very reasonable solution.

>>
>> Sorry I don't follow. =C2=A0Some sample code would be helpful.


>I would assume something along the lines of (untested):


>from pickle import dump


>MOD_THRESHOLD =3D 42


>class PersistentList(list):
> def __init__(self, filepath):
> self.filepath =3D filepath
> self._mod_count =3D 0


> def save(self):
> with open(self.filepath, 'w') as f:
> dump(self, f)
> self._mod_count =3D 0


> def append(self, *args, **kwds):
> super(PersistentList, self).append(*args, **kwds)
> self._mod_count +=3D 1
> if self._mod_count >=3D MOD_THRESHOLD:
> # obviously time-since-last-dump or other
> # more sophisticated metrics might be used instead
> self.save()


Even though it is saved periodically to disk, it looks like the
whole list remains in memory all the time? (If so, it's not what
I'm looking for; the whole point of saving stuff to disk is to keep
the list's memory footprint low.)

~K
 
Reply With Quote
 
kj
Guest
Posts: n/a
 
      08-15-2010
In <i486al$bk$(E-Mail Removed)> "Martin v. Loewis" <(E-Mail Removed)> writes:

>> Does anyone know of such a module?


>ZODB supports persistent lists.


Thanks; I'll check it out.

~K
 
Reply With Quote
 
Dave Angel
Guest
Posts: n/a
 
      08-15-2010
kj wrote:
> <snip>
>> self.save()
>>

>
> Even though it is saved periodically to disk, it looks like the
> whole list remains in memory all the time? (If so, it's not what
> I'm looking for; the whole point of saving stuff to disk is to keep
> the list's memory footprint low.)
>
> ~K
>
>

It sounds like we all made the wrong assumption about your
requirements. It's not persistence you want, but low memory footprint
? Do you or don't you also want the data to survive multiple runs of
the program?

If you want the data to be in a persistent disk file, you probably
should use a database. And if you don't, then you could either increase
the size of your swapfile, or use a more complex scheme to do a memory
mapped file while the program is running, and delete it on exit. The
swapfile is likely to be execute more quickly than anything else you
could cook up in pure python.

DaveA

 
Reply With Quote
 
Emile van Sebille
Guest
Posts: n/a
 
      08-15-2010
On 8/15/2010 3:58 AM kj said...
> In<i486al$bk$(E-Mail Removed)> "Martin v. Loewis"<(E-Mail Removed)> writes:
>
>>> Does anyone know of such a module?

>
>> ZODB supports persistent lists.

>
> Thanks; I'll check it out.
>


I wouldn't expect a low memory footprint however.

Emile



 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off




Advertisments