Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Pickling classes (not class instances)

Reply
Thread Tools

Pickling classes (not class instances)

 
 
Nicolas M. Thiéry
Guest
Posts: n/a
 
      01-10-2009
Dear python developers,

Purpose of this e-mail:
-----------------------

How to customize how a class (not an instance of a class!!!) is
pickled?


Example:
================================================== ============================
class metaclass(type):
def __new__(mcs, name, bases, dict):
print " running metaclass.__new__"
return type.__new__(mcs, name, bases, dict)
#
def __reduce_class__(self):
print "reducing a class"
# do the real work

c = metaclass("foo", (object,), dict())

import copy_reg
copy_reg.pickle(metaclass, metaclass.__reduce_class__)

pickle.dumps(c)
---------------------------------------------------------------------------
PicklingError Traceback (most recent call
last)
....
PicklingError: Can't pickle <class '__main__.foo'>: it's not found as
__main__.foo
================================================== ============================


Context:
--------

I am working on the Sage project (www.sagemath.org), and more
precisely on the category infrastructure. The code is building lots of
classes on the fly by composing preexisting classes by inheritance
(for the curious, see the doc of the class Category in
http://sage.math.washington.edu:2144...ries-nt.patch)..

Since those classes are built on the fly, they cannot be pickled with
the default mechanism of name lookup. A proper pickling would be to
rebuild the class anew. Nothing problematic, except for the question
above.


Discussion:
-----------

It sounds like copy_reg would be the natural way to go (as in the
example above). However, its documentation suggests that it explicitly
is not intended for pickling classes, e.g. first paragraph of:

http://docs.python.org/library/copy_...odule-copy_reg

is:

The copy_reg module provides support for the pickle and cPickle
modules. The copy module is likely to use this in the future as
well. It provides configuration information about object
constructors which are not classes. Such constructors may be factory
functions or class instances.

And indeed, looking up at the code of pickle (l. 289-299 of pickle.py)
(and similarly in cpickle), the copy-reg dispatch is explicit bypassed
for metaclasses:


# Check for a class with a custom metaclass; treat as regular
class
try:
issc = issubclass(t, TypeType)
except TypeError: # t is not a class (old Boost; see SF
#502085)
issc = 0
if issc:
self.save_global(obj)
return

# Check copy_reg.dispatch_table
reduce = dispatch_table.get(t)

Which causes the failure above.


Is there a specific reason for this restriction?

Would it be thinkable to move up the copy reg dispatch before the
metaclass treatment in pickle and cPickle? I did it locally, and it
fixed my problem.

If not, what would be the right way to achieve this?

Many thanks in advance!

Best regards,
Nicolas
--
Nicolas M. Thiéry "Isil" <(E-Mail Removed)>
http://Nicolas.Thiery.name/
 
Reply With Quote
 
 
 
 
Nicolas M. Thiéry
Guest
Posts: n/a
 
      02-14-2009
Dear python developers,

I got no answer to my previous post on this thread "Pickling classes
(not class instances)".
This issue is a show stopper for our project. Any suggestion for where
to ask?

Thanks in advance!

Best regards,
Nicolas

> Purpose of this e-mail:
> -----------------------
>
> How to customize how a class (not an instance of a class!!!) is
> pickled?
>
> Example:
> ================================================== ============================
> class metaclass(type):
> def __new__(mcs, name, bases, dict):
> print " running metaclass.__new__"
> return type.__new__(mcs, name, bases, dict)
> #
> def __reduce_class__(self):
> print "reducing a class"
> # do the real work
>
> c = metaclass("foo", (object,), dict())
>
> import copy_reg
> copy_reg.pickle(metaclass, metaclass.__reduce_class__)
>
> pickle.dumps(c)
> ---------------------------------------------------------------------------
> PicklingError Traceback (most recent call
> last)
> ...
> PicklingError: Can't pickle <class '__main__.foo'>: it's not found as
> __main__.foo
> ================================================== ============================
>
> Context:
> --------
>
> I am working on the Sage project (www.sagemath.org), and more
> precisely on the category infrastructure. The code is building lots of
> classes on the fly by composing preexisting classes by inheritance
> (for the curious, see the doc of the class Category inhttp://sage.math.washington.edu:2144/file/1567cea09170/categories-nt....).
>
> Since those classes are built on the fly, they cannot be pickled with
> the default mechanism of name lookup. A proper pickling would be to
> rebuild the class anew. Nothing problematic, except for the question
> above.
>
> Discussion:
> -----------
>
> It sounds like copy_reg would be the natural way to go (as in the
> example above). However, its documentation suggests that it explicitly
> is not intended for pickling classes, e.g. first paragraph of:
>
> http://docs.python.org/library/copy_...odule-copy_reg
>
> is:
>
> The copy_reg module provides support for the pickle and cPickle
> modules. The copy module is likely to use this in the future as
> well. It provides configuration information about object
> constructors which are not classes. Such constructors may be factory
> functions or class instances.
>
> And indeed, looking up at the code of pickle (l. 289-299 of pickle.py)
> (and similarly in cpickle), the copy-reg dispatch is explicit bypassed
> for metaclasses:
>
> # Check for a class with a custom metaclass; treat as regular
> class
> try:
> issc = issubclass(t, TypeType)
> except TypeError: # t is not a class (old Boost; see SF
> #502085)
> issc = 0
> if issc:
> self.save_global(obj)
> return
>
> # Check copy_reg.dispatch_table
> reduce = dispatch_table.get(t)
>
> Which causes the failure above.
>
> Is there a specific reason for this restriction?
>
> Would it be thinkable to move up the copy reg dispatch before the
> metaclass treatment in pickle and cPickle? I did it locally, and it
> fixed my problem.
>
> If not, what would be the right way to achieve this?
>
> Many thanks in advance!
>
> Best regards,
> Nicolas
> --
> Nicolas M. Thiéry "Isil" <(E-Mail Removed)>http://Nicolas.Thiery.name/


 
Reply With Quote
 
 
 
 
Aaron Brady
Guest
Posts: n/a
 
      02-14-2009
On Feb 14, 2:19*am, Nicolas M. Thiéry <(E-Mail Removed)>
wrote:
> * * * * *Dear python developers,
>
> I got no answer to my previous post on this thread "Pickling classes
> (not class instances)".
> This issue is a show stopper for our project. Any suggestion for where
> to ask?

snip to
http://groups.google.com/group/comp....9304aa0cc791c2
> > How to customize how a class (not an instance of a class!!!) is
> > pickled?

snip
> > Is there a specific reason for this restriction?

>
> > Would it be thinkable to move up the copy reg dispatch before the
> > metaclass treatment in pickle and cPickle? I did it locally, and it
> > fixed my problem.

>
> > If not, what would be the right way to achieve this?


Hi Nicolas,

I couldn't get yours to work. I also tried defining '__reduce__' and
'__getstate__' in the metaclass; they didn't work either.

Can you just subclass Pickler, and override the 'save' method?

> > Is there a specific reason for this restriction?


It sounds very odd. I'm not a core dev, so I can't say.

I also tried defining '__subclasshook__' to convince 'metaclass' it is
not a subclass of 'type'. It didn't work, but 'register' was not
working the way I wanted.

Perhaps you could define your own 'custom_type' type, which looks and
acts just like type, but just doesn't pass the 'subclass' test. It
may be as simple as just cutting-and-pasting the TypeObject definition
from the core.


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Pickling classes -- disappearing lists? Aaron Scott Python 2 07-13-2009 11:00 PM
Pickling dynamically generated classes George Sakkis Python 0 01-26-2008 07:48 AM
Definitive documentation on newstyle classes? (WAS: Pickling andinheritance are making me hurt) Daniel Bickett Python 2 02-06-2005 01:39 AM
Re: Pickling new-style classes w/ getinitargs Dave Brueck Python 0 02-10-2004 10:05 PM
Pickling new-style classes w/ getinitargs Dave Brueck Python 0 02-10-2004 10:01 PM



Advertisments