Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > pickling subclasses of dict/list

Reply
Thread Tools

pickling subclasses of dict/list

 
 
Edward Loper
Guest
Posts: n/a
 
      07-03-2004
I'm having trouble pickling subclasses of dict when they contain cycles.
In particular:

>>> import pickle
>>> class D(dict): pass
>>> d = D()
>>> d[1] = d # add a cycle.
>>> print d

{1: {...}}
>>> pickle.dump(d, open('d.pickle', 'w'))

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/tmp/python-rSB6hN", line 6, in ?
pickle.dump(d, open('d.pickle', 'w'))
File "/usr/lib/python2.3/pickle.py", line 1382, in dump
Pickler(file, protocol, bin).dump(obj)
[...]
File "/usr/lib/python2.3/pickle.py", line 414, in save_reduce
save(func)
RuntimeError: maximum recursion depth exceeded

Note that pickling dict's that contain cycles works just fine, as does
pickling classes that contain cycles in their instance variables. E.g.:

>>> d = {}
>>> d[1] = d
>>> print d
>>> pickle.dump(d, open('d.pickle', 'w'))
>>> print pickle.load(open('d.pickle'))

{1: {...}}

I tried several things with __getstate__, __reduce__, etc., but couldn't
get this to work. Is there some magic that I'm missing? What's the
best way to get around this? (And should I file this as a bug in
pickle? Or am I just not seeing the right way to do it?)

-Edward

 
Reply With Quote
 
 
 
 
Greg Chapman
Guest
Posts: n/a
 
      07-04-2004
On Sat, 03 Jul 2004 14:37:28 -0400, Edward Loper
<(E-Mail Removed)> wrote:

>I'm having trouble pickling subclasses of dict when they contain cycles.
>
>Note that pickling dict's that contain cycles works just fine, as does
>pickling classes that contain cycles in their instance variables. E.g.:
>
> >>> d = {}
> >>> d[1] = d
> >>> print d
> >>> pickle.dump(d, open('d.pickle', 'w'))
> >>> print pickle.load(open('d.pickle'))

> {1: {...}}
>
>I tried several things with __getstate__, __reduce__, etc., but couldn't
>get this to work. Is there some magic that I'm missing? What's the
>best way to get around this? (And should I file this as a bug in
>pickle? Or am I just not seeing the right way to do it?)
>


I think the answer to this problem is to use the extended __reduce__ API
documented here:

http://www.python.org/peps/pep-0307.html

Your subclass's __reduce__ can return a dictitems (or listitems) iterator, the
items of which will get pickled after the subclass instance is memoized, thus
avoiding the infinite recursion. The following works for your simple example:

>>> class D(dict):

.... def __reduce__(self):
.... return (D, (), None, None, self.iteritems())
....
>>> d = D()
>>> d[1] = d
>>> pickle.dumps(d)

'c__main__\nD\np0\n(tRp1\nI1\ng1\ns.'
>>> s = _
>>> dd = pickle.loads(s)
>>> dd.items()

[(1, {1: {...}})]

---
Greg Chapman

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Pickling an extension type subclasses problems Gaetan de Menten Python 0 02-05-2010 02:25 PM
Pickling object inherited from dict Thomas Guettler Python 1 10-28-2003 09:04 AM
Pickling/unpickling extensions types Bob Python 1 09-11-2003 06:13 AM
PS: Help pickling across sockets Jonathan Hayward Python 1 08-17-2003 02:39 AM
Help pickling across sockets Jonathan Hayward Python 0 08-17-2003 12:51 AM



Advertisments