Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > FAQ: how to vary the byte offset of a field of a ctypes.Structure

Reply
Thread Tools

FAQ: how to vary the byte offset of a field of a ctypes.Structure

 
 
p.lavarre@ieee.org
Guest
Posts: n/a
 
      05-31-2007
How do I vary the byte offset of a field of a ctypes.Structure?

How do I "use the dynamic nature of Python, and (re-)define the data
type after the required size is already known, on a case by case
basis"?

\\\

For example, suppose sometimes I receive the value '\x03hi' + \x04bye'
for the struct:

class Struct34(ctypes.Structure):
_pack_ = 1
_fields_ = [('first', 3 * ctypes.c_ubyte),
('second', 4 * ctypes.c_ubyte)]

but then sometimes instead I receive the value '\x05left' + \x06right'
for the struct:

class Struct56(ctypes.Structure):
_pack_ = 1
_fields_ = [('first', 5 * ctypes.c_ubyte),
('second', 6 * ctypes.c_ubyte)]

Thus in general I receive (0xFF ** 2) possible combinations of field
lengths.

///

How do I declare all those hugely many simply regular combinations as
one CTypes.structure?

I also need to do series of 3 or 4 or 5 strings, not just 2 strings.
But always the byte offsets of the subsequent fields vary when the
byte sizes of the preceding fields vary. The byte size of the
enclosing packed struct varies as the length of the packed bytes it
contains.

The errors I get as I try techniques that don't work include:

AttributeError: '_fields_' must be a sequence of pairs
AttributeError: _fields_ is final
ValueError: Memory cannot be resized because this object doesn't own
it
TypeError: incompatible types, c_ubyte_Array_2 instance instead of
c_ubyte_Array_1 instance

How do I change the offset of a field of a ctypes.Structure?

Is the answer to contribute to the next version of CTypes? Or is this
feature already supported somehow?

Curiously yours, thank in advance,

http://www.google.com/search?q=ctypes+variable+size
http://www.google.com/search?q=ctypes+variable+length
http://www.google.com/search?q=ctypes+variable+offset
http://www.google.com/search?q=ctype...+of+strings%22
http://www.google.com/search?q=ctype...byte+offset%22
http://www.google.com/search?q=ctype...byte+offset%22

 
Reply With Quote
 
 
 
 
Larry Bates
Guest
Posts: n/a
 
      05-31-2007
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> How do I vary the byte offset of a field of a ctypes.Structure?
>
> How do I "use the dynamic nature of Python, and (re-)define the data
> type after the required size is already known, on a case by case
> basis"?
>
> \\\
>
> For example, suppose sometimes I receive the value '\x03hi' + \x04bye'
> for the struct:
>
> class Struct34(ctypes.Structure):
> _pack_ = 1
> _fields_ = [('first', 3 * ctypes.c_ubyte),
> ('second', 4 * ctypes.c_ubyte)]
>
> but then sometimes instead I receive the value '\x05left' + \x06right'
> for the struct:
>
> class Struct56(ctypes.Structure):
> _pack_ = 1
> _fields_ = [('first', 5 * ctypes.c_ubyte),
> ('second', 6 * ctypes.c_ubyte)]
>
> Thus in general I receive (0xFF ** 2) possible combinations of field
> lengths.
>
> ///
>
> How do I declare all those hugely many simply regular combinations as
> one CTypes.structure?
>
> I also need to do series of 3 or 4 or 5 strings, not just 2 strings.
> But always the byte offsets of the subsequent fields vary when the
> byte sizes of the preceding fields vary. The byte size of the
> enclosing packed struct varies as the length of the packed bytes it
> contains.
>
> The errors I get as I try techniques that don't work include:
>
> AttributeError: '_fields_' must be a sequence of pairs
> AttributeError: _fields_ is final
> ValueError: Memory cannot be resized because this object doesn't own
> it
> TypeError: incompatible types, c_ubyte_Array_2 instance instead of
> c_ubyte_Array_1 instance
>
> How do I change the offset of a field of a ctypes.Structure?
>
> Is the answer to contribute to the next version of CTypes? Or is this
> feature already supported somehow?
>
> Curiously yours, thank in advance,
>


How about something like:

class fooStruct(ctypes.Structure):
_pack_ = 1
_fields_=[]
def __init__(self, fields):
self._fields_=fields
ctypes.Structure.__init__(self)

a=fooStruct([('first', 3*ctypes.c_ubyte),
('second', 4*ctypes.c_ubyte)])

print a._fields_


-Larry
 
Reply With Quote
 
 
 
 
Thomas Heller
Guest
Posts: n/a
 
      05-31-2007
(E-Mail Removed) schrieb:
> How do I vary the byte offset of a field of a ctypes.Structure?
>
> How do I "use the dynamic nature of Python, and (re-)define the data
> type after the required size is already known, on a case by case
> basis"?
>
> \\\
>
> For example, suppose sometimes I receive the value '\x03hi' + \x04bye'
> for the struct:
>
> class Struct34(ctypes.Structure):
> _pack_ = 1
> _fields_ = [('first', 3 * ctypes.c_ubyte),
> ('second', 4 * ctypes.c_ubyte)]
>
> but then sometimes instead I receive the value '\x05left' + \x06right'
> for the struct:
>
> class Struct56(ctypes.Structure):
> _pack_ = 1
> _fields_ = [('first', 5 * ctypes.c_ubyte),
> ('second', 6 * ctypes.c_ubyte)]
>
> Thus in general I receive (0xFF ** 2) possible combinations of field
> lengths.
>
> ///
>
> How do I declare all those hugely many simply regular combinations as
> one CTypes.structure?
>
> I also need to do series of 3 or 4 or 5 strings, not just 2 strings.
> But always the byte offsets of the subsequent fields vary when the
> byte sizes of the preceding fields vary. The byte size of the
> enclosing packed struct varies as the length of the packed bytes it
> contains.


Often it helps to ask yourself the question: How would I do this in C?

IMO, the answer to this question, applied to your problem, would be:
*Not* by using a structure. A structure is fine if the definition is fixed,
or at most has *one* variable sized field at the very end. Nothing
is true for your problem.

Thomas

 
Reply With Quote
 
p.lavarre@ieee.org
Guest
Posts: n/a
 
      05-31-2007
ctypes.sizeof(a) is still zero, as if ctypes.Structure.__init__
fetches a.__class__._fields_ rather than a._fields_

 
Reply With Quote
 
p.lavarre@ieee.org
Guest
Posts: n/a
 
      05-31-2007
""" Thomas,

Ouch ouch I must have misunderstood what you meant by "use the dynamic
nature of Python, and (re-)define the data type after the required
size is already known, on a case by case basis".

Do you have an example of what you meant? I searched but did not find.
Are those your words?

Yes, to declare strings of strings in Python would be to express a
familiar idea that I don't know how to say well in C.

These are standard structs that I exchange with third parties, e.g.,
me editing Class files read later by a Java interpreter, so I can't
avoid having to deal with this complexity designed into them. For
discussion I've simplified the problem: back in real life I have a few
dozen variations of structs like this to deal with.

The following Python mostly works, but only at the cost of rewriting
the small part of CTypes that I need for this app.
"""

import binascii
import struct

class Struct:

def __init__(self, declarations = []):

"""Add initial values to self and list the names declared."""

names = []
for (initial, name) in declarations:
names += [name]
python = ' '.join(['self.' + name, '=',
'initial'])
exec python
self._names_ = names

def _fields_(self):

"""List the fields."""

fields = []
for name in self._names_:
python = 'self.' + name
fields += [eval(python)]
return fields

def _pack_(self):

"""Pack a copy of the fields."""

packs = ''
for field in self._fields_():
packs += field._pack_()
return packs

def _size_(self, bytes = None):

"""Count the bytes of the fields."""

return len(self._pack_())

def _unpack_(self, bytes):

"""Count the bytes of a copy of the fields."""

offset = 0
for field in self._fields_():
size = field._size_(bytes[offset:])
field._unpack_(bytes[offset:][:size])
offset += size
if offset != len(bytes):
why = ('_unpack_ requires a string argument'
'of length %d' % offset)
raise struct.error(why)

class Field(Struct):

"""Contain one value."""

def __init__(self, value = 0):
self._value_ = value

def _pack_(self):
raise AttributeError('abstract')

def _unpack_(self, bytes):
raise AttributeError('abstract')

class Byte(Field):

"""Contain one byte."""

def _pack_(self):
return struct.pack('B', self._value_)

def _unpack_(self, bytes):
self._value_ = struct.unpack('B', bytes)[0]

class ByteArray(Field):

"""Contain the same nonnegative number of bytes always."""

def _pack_(self):
return self._value_

def _unpack_(self, bytes):
if len(bytes) == len(self._value_):
self._value_ = bytes
else:

why = ('_unpack_ requires a string argument'
'of length %d' % len(self._value_))
raise struct.error(why)

class Symbol(Struct):

"""Contain a count of bytes."""

def __init__(self, value = ''):
Struct.__init__(self, [
(Byte(), 'length'),
(ByteArray(value), 'bytes')])
self._pack_()


def _size_(self, bytes = None):
return ord(bytes[0])

def _pack_(self):
self.length = Byte(self.length._size_() +
self.bytes._size_())
return Struct._pack_(self)

class TwoSymbols(Struct):

"""Contain two Symbols."""

def __init__(self, values = ['', '']):
Struct.__init__(self, [
(Symbol(values[0]), 'first'),
(Symbol(values[1]), 'second')])

zz = Symbol()
print binascii.hexlify(zz._pack_()).upper()
# 01

zz = Symbol()
zz.bytes = ByteArray('left')
zz._unpack_(zz._pack_()) ; print repr(zz._pack_())
# '\x05left'

zz = Symbol()
zz.bytes = ByteArray('right')
zz._unpack_(zz._pack_()) ; print repr(zz._pack_())
# '\x06right'

zz = TwoSymbols()
zz.first.bytes = ByteArray('hi')
zz.second.bytes = ByteArray('bye')
zz._unpack_(zz._pack_()) ; print repr(zz._pack_())
# '\x03hi\x04bye'

print zz._size_()
# 7

yy = '''
def yyFunc(self):
return [self.length, self.bytes]
'''
exec yy
Symbol._fields_ = yyFunc
zz = TwoSymbols(['alef', 'bet'])
zz._unpack_(zz._pack_()) ; print repr(zz._pack_())

 
Reply With Quote
 
p.lavarre@ieee.org
Guest
Posts: n/a
 
      06-01-2007
I see that changing self._fields_ doesn't change ctypes.sizeof(self).

I guess ctypes.Structure.__init__(self) fetches
self.__class__._fields_ not self._fields_.

 
Reply With Quote
 
p.lavarre@ieee.org
Guest
Posts: n/a
 
      06-02-2007
> Often it helps to ask yourself the question: How would I do this in C? ...
>
> *Not* by using a structure. A structure is fine if the definition is fixed,
> or at most has *one* variable sized field at the very end.


http://docs.python.org/lib/module-pickle.html
might be near where I'll find concise Python ways of pickling and
unpickling the (0xFF ** N) possible ways of packing N strings of byte
lengths of 0..xFE together ...

I notice by now I need what ctypes does well often enough that I have
to make my corrupt copy of a subset of ctypes coexist with ctypes per
se. So I have two different BYTE definitions. One is the
ctypes.c_ubyte. The other is my corrupt copy. And likewise for a big-
endian BIG_WORD and so on. So I end up naming the one BYTE and the
other BITE, the one BIG_WORD and the other BIG_WURD, yuck.

 
Reply With Quote
 
p.lavarre@ieee.org
Guest
Posts: n/a
 
      06-04-2007
> > http://docs.python.org/lib/module-pickle.html
> > ... concise Python ways of pickling and unpickling
> > the (0xFF ** N) possible ways of
> > packing N strings of byte lengths of 0..xFE together ...


Aye, looks like an exercise left open for the student to complete:

>>> pickle.dumps("")

"S''\np0\n."
>>>
>>> pickle.dumps("abc")

"S'abc'\np0\n."
>>>
>>> pickle.loads(pickle.dumps("abc"))

'abc'
>>>
>>> pickle.dumps(ctypes.c_ubyte(0))

....
TypeError: abstract class
>>>


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
Comparing offset-aware and offset-naive datetimes? Roy Smith Python 4 01-27-2013 03:17 PM
finding the byte offset of an element in an XML file (tell() and seek()?) Ben Temperton Python 1 06-14-2012 08:28 PM
executing a file with a byte offset? (pipe?) Xilo Musimene C++ 1 01-13-2004 09:21 PM
Translated Offset to Source Offset Lance Riedel XML 2 10-15-2003 03:04 PM



Advertisments