Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > writing pickle function

Reply
Thread Tools

writing pickle function

 
 
perfreem@gmail.com
Guest
Posts: n/a
 
      01-23-2009
hello,

i am using nested defaultdict from collections and i would like to
write it as a pickle object to a file. when i try:

from collections import defaultdict
x = defaultdict(lambda: defaultdict(list))

and then try to write to a pickle file, it says:

TypeError: can't pickle function objects

is there a way around this? it's simply a dictionary that i want to
write to file.. this works no problems with ordinary dicts.

thank you.
 
Reply With Quote
 
 
 
 
Chris Rebert
Guest
Posts: n/a
 
      01-23-2009
On Fri, Jan 23, 2009 at 6:48 AM, <(E-Mail Removed)> wrote:
> hello,
>
> i am using nested defaultdict from collections and i would like to
> write it as a pickle object to a file. when i try:
>
> from collections import defaultdict
> x = defaultdict(lambda: defaultdict(list))
>
> and then try to write to a pickle file, it says:
>
> TypeError: can't pickle function objects
>
> is there a way around this? it's simply a dictionary that i want to
> write to file.. this works no problems with ordinary dicts.


Functions aren't pickleable because they're Python code (which is
itself not pickleable). defaultdicts contain a reference to a function
(in your case, a function defined using lambda), which they use to
create default values. Thus, this causes defaultdicts to not be
pickleable.

This is easily worked around by pickling a plain dict w/ the contents
of the defaultdict (i.e. dict(x) ) and then doing:

x = defaultdict(lambda: defaultdict(list)) #create empty defaultdict
x.update(pickle.load(the_file)) #shove contents of pickled dict into
the defaultdict

Cheers,
Chris

--
Follow the path of the Iguana...
http://rebertia.com
 
Reply With Quote
 
 
 
 
Gerard Flanagan
Guest
Posts: n/a
 
      01-23-2009
On Jan 23, 2:48*pm, (E-Mail Removed) wrote:
> hello,
>
> i am using nested defaultdict from collections and i would like to
> write it as a pickle object to a file. when i try:
>
> from collections import defaultdict
> x = defaultdict(lambda: defaultdict(list))
>
> and then try to write to a pickle file, it says:
>
> TypeError: can't pickle function objects
>
> is there a way around this? it's simply a dictionary that i want to
> write to file.. this works no problems with ordinary dicts.
>
> thank you.



One way via a subclass:

(from memory)

class MyCollection(defaultdict):

def __init__(self):
defaultdict.__init__(self, list)

def __reduce__(self):
return (MyCollection, (), None, None, self.iteritems())

and if you are so inclined (nothing to do with pickling):

__setattr__ = defaultdict.__setitem__
__getattr__ = defaultdict.__getitem__

G.
 
Reply With Quote
 
Peter Otten
Guest
Posts: n/a
 
      01-23-2009
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:

> i am using nested defaultdict from collections and i would like to
> write it as a pickle object to a file. when i try:
>
> from collections import defaultdict
> x = defaultdict(lambda: defaultdict(list))
>
> and then try to write to a pickle file, it says:
>
> TypeError: can't pickle function objects
>
> is there a way around this? it's simply a dictionary that i want to
> write to file.. this works no problems with ordinary dicts.


The error message is misleading. You can pickle your defaultdict if you use
a "normal" function instead of the lambda:

$ cat pickle_defaultdict.py
import sys
from collections import defaultdict
from cPickle import dumps, loads

if "--lambda" in sys.argv:
make_inner = lambda: defaultdict(list)
else:
def make_inner():
return defaultdict(list)

d = defaultdict(make_inner)
d[1][2].append(42)
e = loads(dumps(d))
print e == d

$ python pickle_defaultdict.py
True
$ python pickle_defaultdict.py --lambda
Traceback (most recent call last):
File "pickle_defaultdict.py", line 13, in <module>
e = loads(dumps(d))
File "/usr/lib/python2.5/copy_reg.py", line 69, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects

Peter
 
Reply With Quote
 
Steve Holden
Guest
Posts: n/a
 
      01-23-2009
Peter Otten wrote:
> (E-Mail Removed) wrote:
>
>> i am using nested defaultdict from collections and i would like to
>> write it as a pickle object to a file. when i try:
>>
>> from collections import defaultdict
>> x = defaultdict(lambda: defaultdict(list))
>>
>> and then try to write to a pickle file, it says:
>>
>> TypeError: can't pickle function objects
>>
>> is there a way around this? it's simply a dictionary that i want to
>> write to file.. this works no problems with ordinary dicts.

>
> The error message is misleading. You can pickle your defaultdict if you use
> a "normal" function instead of the lambda:
>
> $ cat pickle_defaultdict.py
> import sys
> from collections import defaultdict
> from cPickle import dumps, loads
>
> if "--lambda" in sys.argv:
> make_inner = lambda: defaultdict(list)
> else:
> def make_inner():
> return defaultdict(list)
>
> d = defaultdict(make_inner)
> d[1][2].append(42)
> e = loads(dumps(d))
> print e == d
>
> $ python pickle_defaultdict.py
> True
> $ python pickle_defaultdict.py --lambda
> Traceback (most recent call last):
> File "pickle_defaultdict.py", line 13, in <module>
> e = loads(dumps(d))
> File "/usr/lib/python2.5/copy_reg.py", line 69, in _reduce_ex
> raise TypeError, "can't pickle %s objects" % base.__name__
> TypeError: can't pickle function objects
>

But can you unpickle the objects you pickle? I believe that functions
are references relative to the modules they come from, and so won't be
found unless the same module can be imported at unpickle time. Could be
wrong, haven't read the source, but I understand that functions work the
same as classes.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/

 
Reply With Quote
 
Peter Otten
Guest
Posts: n/a
 
      01-23-2009
Steve Holden wrote:

> Peter Otten wrote:
>> (E-Mail Removed) wrote:
>>
>>> i am using nested defaultdict from collections and i would like to
>>> write it as a pickle object to a file. when i try:
>>>
>>> from collections import defaultdict
>>> x = defaultdict(lambda: defaultdict(list))
>>>
>>> and then try to write to a pickle file, it says:
>>>
>>> TypeError: can't pickle function objects
>>>
>>> is there a way around this? it's simply a dictionary that i want to
>>> write to file.. this works no problems with ordinary dicts.

>>
>> The error message is misleading. You can pickle your defaultdict if you
>> use a "normal" function instead of the lambda:
>>
>> $ cat pickle_defaultdict.py
>> import sys
>> from collections import defaultdict
>> from cPickle import dumps, loads
>>
>> if "--lambda" in sys.argv:
>> make_inner = lambda: defaultdict(list)
>> else:
>> def make_inner():
>> return defaultdict(list)
>>
>> d = defaultdict(make_inner)
>> d[1][2].append(42)
>> e = loads(dumps(d))
>> print e == d
>>
>> $ python pickle_defaultdict.py
>> True
>> $ python pickle_defaultdict.py --lambda
>> Traceback (most recent call last):
>> File "pickle_defaultdict.py", line 13, in <module>
>> e = loads(dumps(d))
>> File "/usr/lib/python2.5/copy_reg.py", line 69, in _reduce_ex
>> raise TypeError, "can't pickle %s objects" % base.__name__
>> TypeError: can't pickle function objects
>>

> But can you unpickle the objects you pickle? I believe that functions
> are references relative to the modules they come from, and so won't be
> found unless the same module can be imported at unpickle time. Could be
> wrong, haven't read the source, but I understand that functions work the
> same as classes.


Indeed, what is stored in the pickle is the module and function name, not
the byte code:

>>> from cPickle import dumps, loads
>>> def f(): print "original"

....
>>> s = dumps(f)
>>> def f(): print "updated"

....
>>> g = loads(s)
>>> g()

updated

That limits pickle as an exchange format to installations that provide
compatible versions of the pickled classes and functions.

Do you think that would be a problem for the OP?

Peter

 
Reply With Quote
 
Steve Holden
Guest
Posts: n/a
 
      01-24-2009
Peter Otten wrote:
> Steve Holden wrote:
>
>> Peter Otten wrote:
>>> (E-Mail Removed) wrote:
>>>
>>>> i am using nested defaultdict from collections and i would like to
>>>> write it as a pickle object to a file. when i try:
>>>>
>>>> from collections import defaultdict
>>>> x = defaultdict(lambda: defaultdict(list))
>>>>
>>>> and then try to write to a pickle file, it says:
>>>>
>>>> TypeError: can't pickle function objects
>>>>
>>>> is there a way around this? it's simply a dictionary that i want to
>>>> write to file.. this works no problems with ordinary dicts.
>>> The error message is misleading. You can pickle your defaultdict if you
>>> use a "normal" function instead of the lambda:
>>>
>>> $ cat pickle_defaultdict.py
>>> import sys
>>> from collections import defaultdict
>>> from cPickle import dumps, loads
>>>
>>> if "--lambda" in sys.argv:
>>> make_inner = lambda: defaultdict(list)
>>> else:
>>> def make_inner():
>>> return defaultdict(list)
>>>
>>> d = defaultdict(make_inner)
>>> d[1][2].append(42)
>>> e = loads(dumps(d))
>>> print e == d
>>>
>>> $ python pickle_defaultdict.py
>>> True
>>> $ python pickle_defaultdict.py --lambda
>>> Traceback (most recent call last):
>>> File "pickle_defaultdict.py", line 13, in <module>
>>> e = loads(dumps(d))
>>> File "/usr/lib/python2.5/copy_reg.py", line 69, in _reduce_ex
>>> raise TypeError, "can't pickle %s objects" % base.__name__
>>> TypeError: can't pickle function objects
>>>

>> But can you unpickle the objects you pickle? I believe that functions
>> are references relative to the modules they come from, and so won't be
>> found unless the same module can be imported at unpickle time. Could be
>> wrong, haven't read the source, but I understand that functions work the
>> same as classes.

>
> Indeed, what is stored in the pickle is the module and function name, not
> the byte code:
>
>>>> from cPickle import dumps, loads
>>>> def f(): print "original"

> ...
>>>> s = dumps(f)
>>>> def f(): print "updated"

> ...
>>>> g = loads(s)
>>>> g()

> updated
>
> That limits pickle as an exchange format to installations that provide
> compatible versions of the pickled classes and functions.
>
> Do you think that would be a problem for the OP?
>

Nice demonstration!

Possibly not, though the original use of lambdas demonstrated at least
confusion. But there needs to be an understanding that the pickled
function has to be importable. Just using a function in __main__ and
then trying to unpickle from another program that doesn't contain the
function won;t hack it.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to pickle a lambda function? Terry Python 1 08-11-2009 08:16 AM
pickle error: can't pickle instancemethod objects Michele Simionato Python 2 05-23-2008 08:29 AM
a pickle's pickle temposs@gmail.com Python 4 08-02-2005 07:20 PM
AssertionError in pickle's memoize function Michael Hohn Python 3 10-31-2004 03:13 PM
pickle function reference ?? ted kelly Python 1 11-12-2003 07:14 AM



Advertisments