Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Code: Rolling a Container Into a String

Reply
Thread Tools

Code: Rolling a Container Into a String

 
 
Kamilche
Guest
Posts: n/a
 
      06-25-2004
I want to convert a dict into string form, then back again. After
discovering that eval is insecure, I wrote some code to roll a Python
object, dict, tuple, or list into a string. I've posted it below. Does
anyone know an easier way to accomplish this? Essentially, I want to
avoid doing an 'eval' on a string to get it back into dict form... but
still allow nested structures. (My current code doesn't handle nested
structures.)

I conked out before writing the 'unroll' code. I'm going to go watch
some boob tube with my husband, instead, and leave that code for
another day. If you know of a better way, or feel like writing the
'unroll' code and posting it, by all means, do so! I'll check
Google newsgroups before I tackle the job, to see if some kind soul
took pity on me.

--Kamilche

import types

SimpleTypes = [types.BooleanType, types.FloatType, types.IntType, \
types.LongType, types.NoneType, types.StringType]

_dictdelim1 = "{"
_dictdelim2 = "}"
_listdelim1 = "["
_listdelim2 = "]"
_tupledelim1 = "("
_tupledelim2 = ")"

def roll(item):
"Return a string representation of an object, dict, tuple, or
list."
return _roll2(item, [], {})


def unroll(s):
"Unrolls a string back into a dict, tuple, or list."
if type(s) != types.StringType:
raise Exception("You may only pass strings to this function!")
err = "Error occurred when parsing " + s + "!"
state = 0
container = None
for c in s:
if c == _dictdelim1:
lookfor = _dictdelim2
elif c == _listdelim1:
lookfor = _listdelim2
elif c == _tupledelim1:
lookfor = _tupledelim2
else:
raise Exception(err)
state = 1



def _quoted(s):
' Return a stringized value'
if type(s) != types.StringType:
return str(s)
else:
l = []
s = s.replace("'", "\'")
l.append("'")
l.append(s)
l.append("'")
return ''.join(l)

def _roll2(d, lst, r):
' Function that does the work.'
# Start of _roll2
t = type(d)
if t == types.DictType:
theid = id(d)
if theid in r:
raise Exception("Recursion detected! Stopping now.")
r[theid] = theid
cnt = 0
lst.append(_dictdelim1)
for key in d.keys():
if key[0] != '_':
lst.append(_quoted(key))
lst.append(': ')
t = type(d[key])
if t in SimpleTypes:
lst.append(_quoted(d[key]))
else:
_roll2(d[key], lst, r)
lst.append(", ")
cnt += 1
if cnt > 0:
del lst[-1]
lst.append(_dictdelim2)
elif t in (types.ListType, types.TupleType):
theid = id(d)
if theid in r:
raise Exception("Recursion detected! Stopping now.")
r[theid] = theid
cnt = 0
if t == types.ListType:
lst.append(_listdelim1)
else:
lst.append(_tupledelim1)
for item in d:
if type(item) in SimpleTypes:
lst.append(_quoted(item))
else:
_roll2(item, lst, r)
lst.append(", ")
cnt += 1
if cnt > 0:
del lst[-1]
if t == types.ListType:
lst.append(_listdelim2)
else:
lst.append(_tupledelim2)
elif hasattr(d, '__dict__'):
_roll2(d.__dict__, lst, r)
else:
raise Exception("Unhandled type " + str(t) + \
"! You may only pass dicts, tuples, lists, and
" + \
"objects with a __dict__ to this function!")
return ''.join(lst)




class simple:
pass

def main():
l = ['List1', 'List2', 'List3']
d = {'Dict1': 'd1', 'Dict2': 'd2', 'list': l}
t = ('Tuple1', d, 'Tuple2')

print "It handles dicts, lists, and tuples."
print roll(t), "\n"

o = simple()
o.name = 'the name'
o.password = 'the password'
o.list = ['ol1', 'ol2']
o.dict = {'od1': None, 'od2': 2}
o.tuple = ('tuple1', 'tuple2')
o.float = 1.5
o.long = 12345678901234567890
o.bool = True
o.int = 1

print "It handles objects."
print roll(o), "\n"

print "It won't roll attributes whose name starts with '_'"
o._recursion = o.tuple
print roll(o), "\n"

print "It will raise an exception if it detects recursion."
print "This next one will cause recursion."
o.recursion = o.tuple
print roll(o), "\n"

print "You can't roll simple types."
print roll('a'), "\n"

if __name__ == "__main__":
main()
 
Reply With Quote
 
 
 
 
Terry Reedy
Guest
Posts: n/a
 
      06-25-2004

"Kamilche" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed) om...
> I want to convert a dict into string form, then back again. After
> discovering that eval is insecure,


With arbitrary code from an arbitrary source, yes.
If you *know* that you are eval-ing your own safe strings, then no problem.

>I wrote some code to roll a Python
> object, dict, tuple, or list into a string.


repr(object) already does that for you. Why duplicate the work?

You only need custom a eval function, which might check that string is safe
(no function calls, no list comps) and then eval, or which might do parsing
and construction itself.

Terry J. Reedy


 
Reply With Quote
 
 
 
 
David Fraser
Guest
Posts: n/a
 
      06-25-2004
Terry Reedy wrote:
> "Kamilche" <(E-Mail Removed)> wrote in message
> news:(E-Mail Removed) om...
>
>>I want to convert a dict into string form, then back again. After
>>discovering that eval is insecure,

>
>
> With arbitrary code from an arbitrary source, yes.
> If you *know* that you are eval-ing your own safe strings, then no problem.
>
>
>>I wrote some code to roll a Python
>>object, dict, tuple, or list into a string.

>
>
> repr(object) already does that for you. Why duplicate the work?
>
> You only need custom a eval function, which might check that string is safe
> (no function calls, no list comps) and then eval, or which might do parsing
> and construction itself.
>
> Terry J. Reedy
>
>

Or use the pprint module which does nice pretty-printing

David
 
Reply With Quote
 
Paul McGuire
Guest
Posts: n/a
 
      06-25-2004
"Kamilche" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed) om...
> I want to convert a dict into string form, then back again. After
> discovering that eval is insecure, I wrote some code to roll a Python
> object, dict, tuple, or list into a string. I've posted it below. Does
> anyone know an easier way to accomplish this? Essentially, I want to
> avoid doing an 'eval' on a string to get it back into dict form... but
> still allow nested structures. (My current code doesn't handle nested
> structures.)
>

unroll is just repr().

Here is a pyparsing routine to "roll up" your data structures, a mere 60
lines or so. It's fairly tolerant of some odd cases, and fully handles
nested data. (Extension to include remaining items, such as boolean data
and scientific notation, is left as an exercise for the reader.) I hope
this is fairly easy to follow - I've dropped in a few comments.

For those of you who've been living in a cave, you can download pyparsing at
http://pyparsing.sourceforge.net.

-- Paul


from pyparsing import Word, ZeroOrMore, OneOrMore, Suppress, Forward, \
quotedString,nums,Combine,Optional,delimitedList,G roup

# create a dictionary of ugly data, complete with nested lists, tuples
# and dictionaries, even imaginary numbers!
d1 = {}
d1['a'] = [1,2,3,[4,5,6]]
d1['b'] = (7,8,(9,10),'a',"",'')
d1['c'] = { 'aa' : 1, 'bb' : "lskdj'slkdjf", 'cc':1.232, 'dd'('z',),) }
d1[('d','e')] = 5+10j

print repr(d1)

testdata = repr(d1)
"""
looks like this:
{'a': [1, 2, 3, [4, 5, 6]], ('d', 'e'): (5+10j), 'c': {'aa': 1, 'cc': 1.232,
'dd': (('z',),), 'bb': "lskdj'slkdjf"}, 'b': (7, 8, (9, 10), 'a', '', '')}
"""

#define low-level data elements
intNum = Word( nums+"+-", nums )
realNum = Combine(intNum + "." + Optional(Word(nums)))
number = realNum | intNum
imagNum = Combine( "(" + number + "+" + number + "j" + ")" )

item = Forward() # set up for recursive grammar definition
tupleDef = Suppress("(") + ( delimitedList( item ) ^
( item + Suppress(",") ) ) + Suppress(")")
listDef = Suppress("[") + delimitedList( item ) + Suppress("]")
keyDef = tupleDef | quotedString | imagNum | number
keyVal = Group( keyDef + Suppress(":") + item )
dictDef = Suppress("{") + delimitedList( keyVal ) + Suppress("}")

item << ( quotedString | number | imagNum |
tupleDef | listDef | dictDef )

# define low-level conversion routines
intNum.setParseAction( lambda s,loc,toks: int(toks[0]) )
realNum.setParseAction( lambda s,loc,toks: float(toks[0]) )
imagNum.setParseAction( lambda s,loc,toks: eval(toks[0]) ) # no built-in to
convert imaginaries?

# strip leading and trailing character from parsed quoted string
quotedString.setParseAction( lambda s,loc,toks: toks[0][1:-1] )

# define list-to-list/tuple/dict routines
evalTuple = lambda s,loc,toks: [ tuple(toks) ]
evalList = lambda s,loc,toks: [ toks.asList() ]
evalDict = lambda s,loc,toks: [ dict([tuple(kv) for kv in toks]) ]

tupleDef.setParseAction( evalTuple )
listDef.setParseAction( evalList )
dictDef.setParseAction( evalDict )

# first element of returned tokens list is the reconstructed list/tuple/dict
results = item.parseString( testdata )[0]
print results

if repr(results) == repr(d1):
print "Eureka!"
else:
print "Compare results for mismatch"
print repr(results)
print repr(d1)


 
Reply With Quote
 
=?iso-8859-15?Q?Pierre-Fr=E9d=E9ric_Caillaud?=
Guest
Posts: n/a
 
      06-25-2004

Use YAML

import yaml

then from your code:

yaml.dump( whatever ) =>

then yaml.loadstring(str)...

It handles objects.
{'name': 'the name', 'tuple': ('tuple1', 'tuple2'), 'int': 1, 'float':
1.5, 'list': ['ol1', 'ol2'], 'long': 12345678901234567890,
'dict': {'od1': None, 'od2': 2}, 'bool': True, 'password': 'the password'}

--- !!__main__.simple # instanciates class for you
bool: True
dict:
od1: ~
od2: 2
float: 1.5
int: 1
list:
- ol1
- ol2
long: 12345678901234567890
name: the name
password: the password
tuple:
- tuple1
- tuple2

It won't roll attributes whose name starts with '_'
{'name': 'the name', 'tuple': ('tuple1', 'tuple2'), 'int': 1, 'float':
1.5, 'list': ['ol1', 'ol2'], 'long': 12345678901234567890,
'dict': {'od1': None, 'od2': 2}, 'bool': True, 'password': 'the password'}

--- !!__main__.simple
_recursion: # yaml does not handle recursion (I have an old version)
- tuple1
- tuple2
bool: True
dict:
od1: ~
od2: 2
float: 1.5
int: 1
list:
- ol1
- ol2
long: 12345678901234567890
name: the name
password: the password
tuple:
- tuple1
- tuple2

It will raise an exception if it detects recursion.
This next one will cause recursion.
--- !!__main__.simple
_recursion:
- tuple1
- tuple2
bool: True
dict:
od1: ~
od2: 2
float: 1.5
int: 1
list:
- ol1
- ol2
long: 12345678901234567890
name: the name
password: the password
recursion:
- tuple1
- tuple2
tuple:
- tuple1
- tuple2
 
Reply With Quote
 
Oliver Pieper
Guest
Posts: n/a
 
      06-25-2004
> I want to convert a dict into string form, then back again.

Since you probably don't want to do this just for the joy of
converting something to string an back ...

Maybe the pickle (and cPickle) module does what you are looking for.

Oliver
 
Reply With Quote
 
Kamilche
Guest
Posts: n/a
 
      06-25-2004
Pierre-Frédéric Caillaud <(E-Mail Removed)> wrote in message news:<opr941cav01v4ijd@musicbox>...

> Use YAML
>

It looked interesting, so I downloaded it... and was confronted with
dozens of files, and the need to compile before use... when I was
looking for a simple cross-platform 2 function solution that didn't
take any DLL's. Dang.

Well, it's a new day, maybe I'll be inspired.
 
Reply With Quote
 
=?iso-8859-15?Q?Pierre-Fr=E9d=E9ric_Caillaud?=
Guest
Posts: n/a
 
      06-25-2004

did you download syck or the pure python yaml parser ?
on Linux the pure python module is just a matter of typing "emerge sync"
but I don't know about Syck...



On 25 Jun 2004 13:36:31 -0700, Kamilche <(E-Mail Removed)> wrote:

> Pierre-Frédéric Caillaud <(E-Mail Removed)> wrote in message
> news:<opr941cav01v4ijd@musicbox>...
>
>> Use YAML
>>

> It looked interesting, so I downloaded it... and was confronted with
> dozens of files, and the need to compile before use... when I was
> looking for a simple cross-platform 2 function solution that didn't
> take any DLL's. Dang.
>
> Well, it's a new day, maybe I'll be inspired.




--
Using Opera's revolutionary e-mail client: http://www.opera.com/m2/
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Rolling String Buffer needed Matt Brooks Ruby 5 10-03-2009 03:34 AM
Copy elements from one STL container to another STL container Marko.Cain.23@gmail.com C++ 4 02-16-2006 05:03 PM
std::transform container => std::abs(container) Steven T. Hatton C++ 4 12-05-2004 07:10 AM
STL: container's values setup by another container Maitre Bart C++ 2 02-11-2004 12:11 AM
std::container::iterator vs std::container::pointer Vivi Orunitia C++ 11 02-04-2004 08:09 AM



Advertisments