«

BartC <> wrote:
> But, once you've used your smemf() to create the data
> representing a packed struct, how do you access a field
> in a the middle?

You don't. This is to format output, so you already know
what you're putting in, and don't need to re-read it.
I'm writing a gif encoder, forkosh.com/gifsave89.html
For a decoder, you would be reading those blocks,
and then your problem is indeed a problem. I'd have
to write the corresponding scan-like function to smemf()
to deal with that.
--
John Forkosh ( mailto: where j=john and f=forkosh )

On Sat, 2012-09-22, JohnF wrote:
> Jorgen Grahn <grahn+> wrote:
....
>> http://docs.python.org/library/struct.html

>> - you give up some speed
>
> Hardly a likely problem. This is typically for i/o,
> not some compute intensive loop. You'd have to be
> formatting god-knows-how-many GB's before anybody
> would notice the overhead.

Now that you mention it, yes, it's probably like that. But I still
suspect that's part of the reason you don't seen that design in real
code. It's a reflex among C programmers: you don't invent interpreted
mini-languages which require parsing at every call.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

Jorgen Grahn <grahn+> writes:
[...]
> Didn't read very carefully, but it's like the Python 'struct' module,
> isn't it?
>
> http://docs.python.org/library/struct.html

It's also reminiscent of Perl's built-in "pack" and "unpack" functions.

http://perldoc.perl.org/functions/pack.html
http://perldoc.perl.org/functions/unpack.html

--
Keith Thompson (The_Other_Keith) kst- <http://www.ghoti.net/~kst>
Will write code for food.
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

"JohnF" <> wrote in message
news:k3klnr$lq7$...
> Jorgen Grahn <grahn+> wrote:

>> Didn't read very carefully, but it's like the Python 'struct' module,
>> isn't it?
>> http://docs.python.org/library/struct.html

> Well, I'm inclined to re-do smemf()'s specs as close as possible
> to python's, except that their format strings are more different
> from C's than necessary. In any case, I'm sure their stuff deals
> with problems that hadn't occurred to me yet. So it'll be a great
> improvement to see precisely what they're doing.

The main problem Python has to deal with is that the language doesn't have
structs.

--
Bartc

On 09/22/12 08:53 PM, JohnF wrote:
> Nick Keighley<> wrote:
>> Eric Sosman<esos...@ieee-dot-org.invalid> wrote:
>>
>> serializing structs
>>
>>> One way is the function-per-struct-type approach, and
>>> although it may be "b..o..r..i..n..g" it has advantages
>>> that should not be dismissed lightly. ?Consider that there
>>> are (most likely) only a handful of structs and hence only
>>> a handful of functions; writing them won't take enough
>>> time to b..o..r..e anyone except an ADHD sufferer.
>>
>> I've dealt with cases where there were considerably more than a
>> "handful"- communication protocol.
>>
>> Since I have a low b..o..r..e..d..o..m threshold I resorted to code
>> generation. The protocol was defined by tables in a PDF document (ug).
>> Copy-paste turned them into text and perl turned them into something
>> easily processed. Most of the code was generated by people who didn't
>> appear to mind writing tons of tedious boring repetitive code. There
>> was a bit-banging library that did most of the heavy lifting.
>
> What do you think of my my smemf()-type of solution in other post?
> Not necessarily that particular solution, but the point being that
> this has got to be a pretty frequently occurring problem, and the
> only available solutions, like yours above, seem to be pretty
> awfully ugly. But it ain't rocket science -- there ought to be
> a way to deal with these situations elegantly. smemf() solves it
> with>>zero<< structs.

having worked on numerous on the wire protocols, I've encountered this
problem many times. The most practical solution for all but the most
trivial cases is to code generate the formatting code. The source for
the code generator can either be C (or C++ if inheritance helps) code or
some other easy to parse format. This is one case where I prefer XML
(often in the form of an OpenOffice document) as the "other easy to
parse format".

--
Ian Collins

Keith Thompson <kst-> wrote:
> Jorgen Grahn <grahn+> writes:
> [...]
>> Didn't read very carefully, but it's like the Python 'struct' module,
>> isn't it?
>>
>> http://docs.python.org/library/struct.html
>
> It's also reminiscent of Perl's built-in "pack" and "unpack" functions.
> http://perldoc.perl.org/functions/pack.html
> http://perldoc.perl.org/functions/unpack.html

Thanks, Keith (and Jorgen again), that's also incredibly useful.
Lots of food for thought to spec out a C variant. And yet another
format/template to peruse. I really should put some effort into
learning these "little languages".
Despite BartC's remark, "The main problem Python has to deal
with is that the language doesn't have structs", I think this
kind of pack function has valuable uses in C (along with a
scan-like unpack, as suggested by BartC's other remark),
e.g., formatting (and reading) binary blocks/packets/whatever,
which is what brought the idea to my mind. And the fact that
all these little languages have pack/unpack just supports
the notion they're useful funcs. Moreover, I'm sure you can
see they're no great big deal to code. So why not do it
(I'm in the process, but that's no reason others shouldn't
do it differently/better/whatever)? The naysayers can just
ignore it. I'm sure everybody has their favorite C feature
they choose not to use.
Finally, regarding Ian's remarks, pack/unpack would be
a >>lightweight<< alternative to accomplish this kind of
task. Ian's way involves importing additional tool dependencies,
possibly turning what could be just a few lines of code
into a subtask unto itself. For his large project case,
involving lots and lots of different block formats,
that may be the best approach (I've worked with Swift
at Bankers Trust, albeit a while back, and also with various
ticker feeds, etc, etc, and more etc). But that's no reason
not to have a lightweight alternative.
--
John Forkosh ( mailto: where j=john and f=forkosh )

JohnF wrote:
> Eric Sosman<> wrote:
>> JohnF wrote:
>>> Any>>portable<< way to accomplish that in c?
>> No.
>>> In particular, I'm trying to write blocks that
>>> conform to a binary file format (gif), and can
>>> set up structs for them easily enough, but can't
>>> fwrite(blockstruct,sizeof(blockstruct),1,fileptr),
>>> or the like, due to blockstruct's inevitable
>>> padding (which indeed occurs for gif format blocks).
>>
>> Aha! You don't need (or want) packed structs at all:
>> You want a way to pluck information from a plain vanilla
>> struct and write it in an externally-defined format.
>> ...
>> You write one super-
>> function that accepts a void* struct pointer and a "struct
>> descriptor," usually an array of byte offsets and type codes:
>> struct struct_descriptor {
>> size_t offset;
>> enum { BYTE, BYTEPAIR, BYTEQUAD, ..., STOP } type; };
>
> Thanks, Eric (and whoever the other guy is). Both your suggestions
> are along the same lines, and, of course, already occurred to me.
> And, as per the drawbacks you pointed out (plus being a pain in
> the neck to maintain two structs per struct), already dismissed
> by me.
>
> I'd actually finished (what I think is) a more elegant solution,
> that I called smemf() that's like memcpy() but under format
> control, including additonal format specifiers for hex, for bits,
> and for other stuff. The code actually works fine, but still
> uncompleted is 723 lines (though that includes>>many<< comments),
> which is somewhat of a tail-wagging-dog situation which I also
> want to avoid.
> The point of smemf is that a single format string replaces
> that entire extra struct. So it's still "extra work for mother",
> but much less in-your-face when reading the program that uses it.
> Since it's not done (and the copyright not registered) yet,
> I'm not releasing it, but (since I can't copyright the idea,
> anyway) below is its (still somewhat incomplete) main comment block
> describing its usage and functional specs, in case anyone else wants
> to take a stab at implementation. Also, some stuff is done but not
> documented, e.g., little/big endian flag to control which way %d works,
> the bit-field specifier, etc. But you'll get the idea. And after that,
> the additional details are obvious to anyone who thinks about it.
>
> /* ================================================== ========================
> * Function: smemf ( unsigned char *mem, char *format, ... )
> * Purpose: Construct a formatted block of memory, typically containing
> * binary data, e.g., network packets, gif images, etc.
> * Behaves much like (a subset of) sprintf, but the intent
> * to accommodate binary data requires a few significant
> * exceptions, as explained in the Notes section below.
> * --------------------------------------------------------------------------
> * Arguments: mem (O) (unsigned char *) to memory block
> * to be formatted.
> * format (I) (char *) to null-terminated string containing
> * specifications for how the variable arg list
> * following it should be formatted in mem.
> * See Notes below.
> * ... as many value args as needed to satisfy
> * the format specification above
> * --------------------------------------------------------------------------
> * Returns: ( int ) # bytes in returned mem,
> * 0 for any error.
> * --------------------------------------------------------------------------
> * Notes: o Like sprintf, mem must already be allocated by caller,
> * and be large enough to accommodate the accompanying
> * format specification argument.
> * o format is sprintf-like, but with some significant exceptions
> * and additions to facilitate smemf's different purpose.
> * The one most significant similarity and difference is...
> * * Like sprintf, conversion specifications are introduced by
> * the % character and terminated by a conversion specifier
> * (see list and discussions below).
> * * But unlike sprintf, ordinary characters occurring
> * outside conversion specifications aren't immediately
> * copied into mem...
> * * Instead, literals that you want formatted in mem
> * must always be followed by corresponding conversion
> * specifications.
> * * For example, 123abc%s formats the next>>six bytes<<
> * of mem with that>>ascii character<< string.
> * But 123abc%x interprets that same string as
> *>>hex digits<<, formatting the next>>three bytes<<
> * of mem accordingly.
> * * And that's why ordinary characters must be followed
> * by a corresponding conversion specification, i.e.,
> * because unlike sprintf, which always interprets ordinary
> * characters as ascii, smemf formats binary memory blocks,
> * too, and therefore needs a conversion specification
> * to interpret the intended meaning of literals.
> * * Additional notes:
> * - When %s isn't preceded by a literal field,
> * then smemf interprets the next argument from your
> * argument list, in the usual way like sprintf.
> * But 123abc%s uses no arguments from your argument list.
> * - Field widths like 123abc%10s generate a 10-byte field,
> * left-justified with your 123abc literal, and
> * right-filled with four blanks. See the field width
> * discussion below for additional information about
> * optional right-justification, non-blank filler, etc.
> * - Leading/trailing whitespace is ignored, so
> * " 123abc %s " is the same as "123abc%s", but any
> * embedded whitespace like "123 abc%s" is respected
> * (although "123 abc%x" would still be an error,
> * while " 123abc %x " becomes okay).
> * - Leading/trailing (or pure) whitespace is obtained
> * by surrounding the literal with its own quotes,
> * e.g., format = " \" 123abc \" %s " includes one blank
> * before and after 123abc.
> * - On the very rare occasion when you want a literal
> * quote character, escape it, i.e., smemf needs
> * to see \" in your format string, so you'd need to
> * write format = " 123\\\"abc %s " to actually format
> * the string 123"abc in mem. That's confusing, but
> * a straightforward application of the obvious rules.
> * o The conversion specifiers recognized by smemf are
> * s,S, x,X, d,D, all discussed in detail below.
> * Note that x,X behave identically, as do d,D,
> * but s,S have different behaviors, discussed in detail below.
> * But first, some general remarks...
> * * s,S,x,X are default left-justified, i.e., the first byte
> * (or first hex digit for x,X) from your literal or argument
> * goes into the next available byte (or hex digit) of mem.
> * But %+etc (i.e., a + flag following %) right-justifies
> * your literal or argument instead, e.g., 123abc%+10s
> * generates a 10-byte field, left-filled with four blanks,
> * then followed by your right-justified 123abc literal.
> * And 123abc%+10x generates a five-byte field, left-filled
> * with two leading 0 bytes, followed by three bytes
> * containing your 12,3A,BC.
> * o The s conversion specifier...
> * *
> * o The S conversion specifier...
> * o The x,X conversion specifier...
> * o The d,D conversion specifier...
> * * A literal 123%d is taken as the decimal integer 123,
> * or the argument for %d is taken as an int.
> * * Justification flags following % are ignored.
> * The bits comprising your int are "right-justified" in
> * your specified field, i.e., low-order bit is rightmost.
> * * A width %10d means 10 bits. But all fields are byte-sized,
> * and 10-bits is "promoted" to 2-bytes, with your int value
> * "right-justified" (explained above) in that 16-bit field.
> * However, if your int value is greater than 1023=2^10-1,
> * then your value is "truncated" to its low-order 10 bits,
> * even though that 16-bit field could accommodate more bits.
> * * If % width.precision d is given, precision is ignored.
> * * If width is not given, it defaults to 16(bits) if your
> * value is less than 65536, or to 32(bits) otherwise.
> * * Example: 47%d generates two bytes containing 00,2F.
> * And 47%17d generates three bytes 00,00,2F.
> * ================================================== ===================== */
>

There is already a library which does very similar stuff.
It is called libtpl.

On 09/23/12 12:19 PM, JohnF wrote:

> Finally, regarding Ian's remarks, pack/unpack would be
> a>>lightweight<< alternative to accomplish this kind of
> task. Ian's way involves importing additional tool dependencies,
> possibly turning what could be just a few lines of code
> into a subtask unto itself.

The up front effort is much the same. In your case you have to code the
pack and unpack functions, in mine I had to write a simple code
generator. In both cases, once it's done it's done. While the code
generator can be simple, mine has evolved to support enum types and
inheritance, but its still only a couple of hundred lines of code. A
colleague on a previous project write all the transforms in XLST, but
that made my head explode.

In coding, I add a table to an OpenOffice document (part of the project
documentation, so it's dual use) while you write a format string.
Probably similar effort although I'd say the table approach is less
error prone.

At run time, I have optimised code to perform the packing and unpacking
while your functions have to interpret the format string each time they
are called. If you have to handle a lot of traffic, especially on a low
powered device, that can be a killer.

--
Ian Collins

Johann Klammer <> wrote:
> JohnF wrote:
>>
>> /* ================================================== =====================
>> * Function: smemf ( unsigned char *mem, char *format, ... )
>> * Purpose: Construct a formatted block of memory, typically containing
>> * binary data, e.g., network packets, gif images, etc.
>> * Behaves much like (a subset of) sprintf, but the intent
>> * to accommodate binary data requires a few significant
>> * exceptions, as explained in the Notes section below.
>> <<snip>>
>
> There is already a library which does very similar stuff.
> It is called libtpl.

Thanks for the pointer, Johann.
http://tpl.sourceforge.net/userguide.html
Also looks interesting, though less similar than pack/unpack
for python and perl, pointed out previously. Nevertheless,
certainly valuable additional food for thought. I'll certainly
study them all, and then try to spec out (what seems to me like)
the best C variant.
--
John Forkosh ( mailto: where j=john and f=forkosh )

Ian Collins <ian-> wrote:
> JohnF wrote:
>
>> Finally, regarding Ian's remarks, pack/unpack would be
>> a lightweight alternative to accomplish this kind of
>> task. Ian's way involves importing additional tool dependencies,
>> possibly turning what could be just a few lines of code
>> into a subtask unto itself.
>
> The up front effort is much the same. <<snip>>

Yeah, well, no biggie. There's no reason not to have
two or several different ways to approach the same problem,
and let the programmer decide what he prefers in a
particular situation. This isn't "either/or", it's "both".
--
John Forkosh ( mailto: where j=john and f=forkosh )