Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > How to better pickle an extension type

Reply
Thread Tools

How to better pickle an extension type

 
 
dgdev
Guest
Posts: n/a
 
      04-16-2007
I would like to pickle an extension type (written in pyrex). I have
it working thus far by defining three methods:

class C:
# for pickling
__getstate__(self):
... # make 'state_obj'
return state_obj

__reduce__(self):
return C,(args,to,__init__),me.__getstate__()

# for unpickling
__setstate__(self,state_obj):
self.x=state_obj.x
...


This gets the class pickling and unpickling.

However, I'd like to not specify arguments for __init__ (as I do now
in __reduce__), and so not have __init__ invoked during unpickling.

I would like to have the pickling machinery somehow create an
uninitialized object, and then call its __setstate__, where I can re-
create it from 'state_obj'.

Is there a kosher way to do so, that is without me having to have a
special mode in the constructor for when the object is being created
by the unpickler?

 
Reply With Quote
 
 
 
 
Alex Martelli
Guest
Posts: n/a
 
      04-17-2007
dgdev <(E-Mail Removed)> wrote:

> I would like to pickle an extension type (written in pyrex). I have
> it working thus far by defining three methods:
>
> class C:
> # for pickling
> __getstate__(self):
> ... # make 'state_obj'
> return state_obj
>
> __reduce__(self):
> return C,(args,to,__init__),me.__getstate__()
>
> # for unpickling
> __setstate__(self,state_obj):
> self.x=state_obj.x
> ...
>
>
> This gets the class pickling and unpickling.
>
> However, I'd like to not specify arguments for __init__ (as I do now
> in __reduce__), and so not have __init__ invoked during unpickling.
>
> I would like to have the pickling machinery somehow create an
> uninitialized object, and then call its __setstate__, where I can re-
> create it from 'state_obj'.
>
> Is there a kosher way to do so, that is without me having to have a
> special mode in the constructor for when the object is being created
> by the unpickler?


I don't understand why you have a problem -- __init__ is NOT called by
default upon loading an object w/__setstate__. Witness:

>>> class C(object):

.... def __init__(self, *a): print 'init', a
.... def __getstate__(self): print 'gs'; return {}
.... def __setstate__(self, *a): print 'ss', a
....
>>> c = C()

init ()
>>> s = cPickle.dumps(c, 2)

gs
>>> s

'\x80\x02c__main__\nC\nq\x01)\x81q\x02}b.'
>>> z = cPickle.loads(s)

ss ({},)


Perhaps you're not using protocol 2? You should be...


Alex
 
Reply With Quote
 
 
 
 
Ziga Seilnacht
Guest
Posts: n/a
 
      04-17-2007
dgdev wrote:
> I would like to pickle an extension type (written in pyrex). I have
> it working thus far by defining three methods:
>
> class C:
> # for pickling
> __getstate__(self):
> ... # make 'state_obj'
> return state_obj
>
> __reduce__(self):
> return C,(args,to,__init__),me.__getstate__()
>
> # for unpickling
> __setstate__(self,state_obj):
> self.x=state_obj.x
> ...
>
> This gets the class pickling and unpickling.
>
> However, I'd like to not specify arguments for __init__ (as I do now
> in __reduce__), and so not have __init__ invoked during unpickling.
>
> I would like to have the pickling machinery somehow create an
> uninitialized object, and then call its __setstate__, where I can re-
> create it from 'state_obj'.
>
> Is there a kosher way to do so, that is without me having to have a
> special mode in the constructor for when the object is being created
> by the unpickler?


Why are you overwriting the __reduce__() method? The default
object.__reduce__() method, inherited by all new style classes,
already does what you want. If you really must overwrite it, and
you don't want __init__() to get called, then you should return a
reconstructor named __newobj__() as the first item of reduce
tuple. Something like this:

>>> def __newobj__(cls, *args):

.... return cls.__new__(cls, *args)
....
>>> class C(object):

.... def __init__(self):
.... print "I shouldn't be called at reconstruction"
.... def __reduce__(self):
.... try:
.... getnewargs = self.__getnewargs__
.... except AttributeError:
.... newargs = (self.__class__,)
.... else:
.... newargs = (self.__class__,) + getnewargs()
.... try:
.... getstate = self.__getstate__
.... except AttributeError:
.... # this ignores __slots__ complications
.... state = self.__dict__
.... else:
.... state = getstate()
.... # this ignores list and dict subclasses
.... return __newobj__, newargs, state
....
>>> c = C()

I shouldn't be called at reconstruction
>>> import pickle
>>> for proto in range(3):

.... assert isinstance(pickle.loads(pickle.dumps(c, proto)), C)
....
>>>


Ziga

 
Reply With Quote
 
dgdev
Guest
Posts: n/a
 
      04-18-2007
Thanks for your replies.

The code I showed above was pyrex code, not python code. You are
correct that python objects do not require .__reduce__() to be
picklable, but apparently c extension types do (makes sense, they must
be more opaque to the python machinery).

I'll try the .__newobj__(), see if I can get it to do what I want...

On Apr 17, 2:50 am, Ziga Seilnacht <(E-Mail Removed)> wrote:
> dgdev wrote:
> > I would like topicklean extension type (written inpyrex). I have
> > it working thus far by defining three methods:

>
> > class C:
> > # for pickling
> > __getstate__(self):
> > ... # make 'state_obj'
> > return state_obj

>
> > __reduce__(self):
> > return C,(args,to,__init__),me.__getstate__()

>
> > # for unpickling
> > __setstate__(self,state_obj):
> > self.x=state_obj.x
> > ...

>
> > This gets the class pickling and unpickling.

>
> > However, I'd like to not specify arguments for __init__ (as I do now
> > in __reduce__), and so not have __init__ invoked during unpickling.

>
> > I would like to have the pickling machinery somehow create an
> > uninitialized object, and then call its __setstate__, where I can re-
> > create it from 'state_obj'.

>
> > Is there a kosher way to do so, that is without me having to have a
> > special mode in the constructor for when the object is being created
> > by the unpickler?

>
> Why are you overwriting the __reduce__() method? The default
> object.__reduce__() method, inherited by all new style classes,
> already does what you want. If you really must overwrite it, and
> you don't want __init__() to get called, then you should return a
> reconstructor named __newobj__() as the first item of reduce
> tuple. Something like this:
>
> >>> def __newobj__(cls, *args):

>
> ... return cls.__new__(cls, *args)
> ...>>> class C(object):
>
> ... def __init__(self):
> ... print "I shouldn't be called at reconstruction"
> ... def __reduce__(self):
> ... try:
> ... getnewargs = self.__getnewargs__
> ... except AttributeError:
> ... newargs = (self.__class__,)
> ... else:
> ... newargs = (self.__class__,) + getnewargs()
> ... try:
> ... getstate = self.__getstate__
> ... except AttributeError:
> ... # this ignores __slots__ complications
> ... state = self.__dict__
> ... else:
> ... state = getstate()
> ... # this ignores list and dict subclasses
> ... return __newobj__, newargs, state
> ...>>> c = C()
>
> I shouldn't be called at reconstruction>>> importpickle
> >>> for proto in range(3):

>
> ... assert isinstance(pickle.loads(pickle.dumps(c, proto)), C)
> ...
>
>
>
> Ziga



 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
Question regarding multiprocessing and error: Can't pickle <type tleeuwenburg@gmail.com Python 1 10-14-2009 05:11 AM
pickle error: can't pickle instancemethod objects Michele Simionato Python 2 05-23-2008 08:29 AM
a pickle's pickle temposs@gmail.com Python 4 08-02-2005 07:20 PM
Build a Better Blair (like Build a Better Bush, only better) Kenny Computer Support 0 05-06-2005 04:50 AM



Advertisments