Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > a pickle's pickle

Reply
Thread Tools

a pickle's pickle

 
 
temposs@gmail.com
Guest
Posts: n/a
 
      08-02-2005
I'm trying to pickle a class, and while I get no errors or anything,
almost none of the class instance gets pickled, and I don't know
why...Here's the pickled output:

(i__main__
TrainingMatrix
p0
(dp1
S'matrixWords'
p2
I4714
sS'numWords'
p3
I4714
sS'totalWordsProcessed'
p4
I46735
sS'numContexts'
p5
I7664
sS'estimator'
p6
(dp7
sb.

--End of output

The class TrainingMatrix has no embedded classes and none of its
methods have embedded methods. An instance of this class running in my
program for about 10 minutes can build up on the order of 100MB in
resident memory, but the output seems to be the same regardless of the
data set size. The output seems to not even capture all of the member
variables in the class. Here is the class code, abridged:

class TrainingMatrix:
matrix = []
estimator = {}
wordInfo = {}
contextInfo = {}
totalWordsProcessed = 0
numWords = 0
numContexts = 0
matrixWords = 0

def AddWordInfo(self,newWordInfo,newCapScheme):
...
#End AddWordInfo

def AddNewWord(self,newCapScheme):
...
#End AddNewCapScheme

def AddContext(self,newContext):
...
#End AddContext

def AddInstance(self,word,context):
...
#End AddInstance

def UpdateMatrix(self,wordIndex,contextIndex,isLowerCa se):
...
#End UpdateMatrix

def PrintMatrix(self):
...
#End PrintMatrix

def EstimateLowerCase(self):
...
#End GetNumWords

def GetWordInfo(self,wordToFind):
...
#End GetWordInfo

def GetContext(self,wordList,direction):
...
#End GetContext

def GetBestCapScheme(self,wordInfo,precedeContext,foll owContext):
...
#End GetBestCapScheme

def IsLowerCase(self,word):
...
#End IsLowerCase

#End TrainingMatrix
###################

And here is the pickling code:

try:
trainDB = open(trainDBString,"r+")
except IOError:
trainDB = open(trainDBString,"w")
trainDB.close()
trainDB = open(trainDBString,"r+")
#End try
....
try:
trainerString = trainDB.read()
trainer = loads(trainerString)

except EOFError:
trainer = TrainingMatrix()
#End try
....
trainerString = dumps(trainer)
trainDB.write(trainerString)

I've also tried a simple shelve implementation but got results similar
to this, which is why I recoded to pickle, since it's lower level. Any
help is appreciated

-Andrew

 
Reply With Quote
 
 
 
 
Magnus Lycka
Guest
Posts: n/a
 
      08-02-2005
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:

> class TrainingMatrix:
> matrix = []
> estimator = {}
> wordInfo = {}
> contextInfo = {}
> totalWordsProcessed = 0
> numWords = 0
> numContexts = 0
> matrixWords = 0


Is there some confusion between the scope of the class
object and the scopes of the instance objects perhaps?

Are you aware of this distinction? See below:

>>> class X:

.... m=0
....
>>> X.m

0
>>> x=X()
>>> x.m

0
>>> x.m += 5
>>> x.m

5
>>> X.m

0

Are you pickling the class object or an instance?

If you are pickling the class: Why? Is the data
really in the class object?

If you are pickling an instance:
Is the data in the class object?
Is the data in another instance object?

 
Reply With Quote
 
 
 
 
temposs@gmail.com
Guest
Posts: n/a
 
      08-02-2005
I intended to pickle the class instance I call trainer...from my code,
also in the first post:

trainerString = trainDB.read()
trainer = loads(trainerString)


except EOFError:
trainer = TrainingMatrix()
....
trainerString = dumps(trainer)
....

So basically trainer always gets an existing TrainingMatrix(the class)
pickled object if there is a file to read from, otherwise it just makes
a new instance. Either way, the instance trainer is pickled at the
end. Maybe I'm missing something...

 
Reply With Quote
 
Benji York
Guest
Posts: n/a
 
      08-02-2005
(E-Mail Removed) wrote:
> So basically trainer always gets an existing TrainingMatrix(the class)
> pickled object if there is a file to read from, otherwise it just makes
> a new instance. Either way, the instance trainer is pickled at the end.


Right, but the data you're interested in is contained in the class, not
the instance. You need to move the mutable class attributes into the
instance. Like so:

class TrainingMatrix:

totalWordsProcessed = 0
numWords = 0
numContexts = 0
matrixWords = 0

def __init__(self):
self.matrix = []
self.estimator = {}
self.wordInfo = {}
self.contextInfo = {}
--
Benji York


 
Reply With Quote
 
temposs@gmail.com
Guest
Posts: n/a
 
      08-02-2005
Benji,
Thanks so much, you have saved the day ^_^

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
pickle error: can't pickle instancemethod objects Michele Simionato Python 2 05-23-2008 08:29 AM
PIX pickle John Caruso Cisco 2 12-13-2003 08:10 AM
import pickle succeeds only after two tries?? Bram Stolk Python 2 09-23-2003 10:25 AM
freeze utility and pickle Aki Niimura Python 1 08-21-2003 03:00 PM
Securing 'pickle' Ben Finney Python 17 07-11-2003 08:30 PM



Advertisments