Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Is there anything that pickle + copy_reg cannot serialize?

Reply
Thread Tools

Re: Is there anything that pickle + copy_reg cannot serialize?

 
 
Jean-Paul Calderone
Guest
Posts: n/a
 
      12-08-2005
On Thu, 08 Dec 2005 22:42:32 +0800, Maurice LING <(E-Mail Removed)> wrote:
>Hi,
>
>I need to look into serialization for python objects, including codes,
>recursive types etc etc. Currently, I have no idea exactly what needs to
>be serialized, so my scope is to be as wide as possible.
>
>I understand that marshal is extended by pickle to serialize class
>instances, shared elements, and recursive data structures
>(http://www.effbot.org/librarybook/pickle.htm) but cannot handle code
>types. pickle can be used together with copy_reg and marshal to
>serialize code types as well
>(http://www.effbot.org/librarybook/copy-reg.htm).
>
>So my question will be, are there anything that pickle/copy_reg/marshal
>combination cannot serialize? If so, what are the workarounds?


Since copy_reg lets you specify arbitrary code to serialize arbitrary
objects, you shouldn't run into any single object that you cannot
serialize to a pickle.

However, both pickle implementations are recursive, so you will be
limited by the amount of memory you can allocate for your stack. By
default, this will limit you to something like object graphs 333 edges
deep or so (if I'm counting stack frames correctly). Note that this
does not mean you cannot serialize more than 333 objects at a time,
merely that if it takes 333 or more steps to go from the first object
to any other object in the graph (using the traversal order pickle
uses), the pickling will fail. You can raise this limit, to a point,
with sys.setrecursionlimit().

Jean-Paul
 
Reply With Quote
 
 
 
 
Maurice LING
Guest
Posts: n/a
 
      12-08-2005

> Since copy_reg lets you specify arbitrary code to serialize arbitrary
> objects, you shouldn't run into any single object that you cannot
> serialize to a pickle.


In http://www.effbot.org/librarybook/pickle.htm, it specifically
mentions that code objects cannot be pickled and require the use of
copy_reg, as follows:

import copy_reg
import pickle, marshal, types

#
# register a pickle handler for code objects

def code_unpickler(data):
return marshal.loads(data)

def code_pickler(code):
return code_unpickler, (marshal.dumps(code),)

copy_reg.pickle(types.CodeType, code_pickler, code_unpickler)

#
# try it out

CODE = """
print "suppose he's got a pointed stick"
"""

code = compile(CODE, "<string>", "exec")

exec code
exec pickle.loads(pickle.dumps(code))


I cannot understand 2 things, which I seek assistance for:
1. Is code object the only thing can cannot be pickled (less facing
recursion limits)?
2. In the above example, how copy_reg works with pickle?

Thanks and Cheers
Maurice

>
> However, both pickle implementations are recursive, so you will be
> limited by the amount of memory you can allocate for your stack. By
> default, this will limit you to something like object graphs 333 edges
> deep or so (if I'm counting stack frames correctly). Note that this
> does not mean you cannot serialize more than 333 objects at a time,
> merely that if it takes 333 or more steps to go from the first object to
> any other object in the graph (using the traversal order pickle uses),
> the pickling will fail. You can raise this limit, to a point, with
> sys.setrecursionlimit().
>
> Jean-Paul

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
problem in running a basic code in python 3.3.0 that includes HTML file Satabdi Mukherjee Python 1 04-04-2013 07:48 PM
pickle error: can't pickle instancemethod objects Michele Simionato Python 2 05-23-2008 08:29 AM
Re: Is there anything that pickle + copy_reg cannot serialize? Jean-Paul Calderone Python 2 12-09-2005 11:23 AM
Is there anything that pickle + copy_reg cannot serialize? Maurice LING Python 0 12-08-2005 02:42 PM
a pickle's pickle temposs@gmail.com Python 4 08-02-2005 07:20 PM



Advertisments