«

On 12-12-20 05:23 AM, wrote:
> On Thursday, December 20, 2012 2:04:17 PM UTC+1, glen herrmannsfeldt wrote:
>>
>>> At first this approach was good, but now I'm finding many problems.
>>> Mostly I have to keep the two applications synchronized: if I add
>>> a parameter (maybe in the middle of the answer, because an array
>>> size is increased) or enlarge a numerical values (from 1 to 2 bytes)
>>> or something else, I have to change both applications.
>>> I usually want to have a retro-compatible master application
>>> (master should communicate well with new or older slaves
>>> application), so the code will be filled with many ifs
>>> (if the slave answer version is 1.0 than there is one parameter
>>> here, if it is 1.2 the array size is 3 and not 2...)
>>
>>> Is there a better approach to exchange data on the wire between
>>> two C applications, considering the limitations of memory,
>>> the low speed and the length of messages of a small embedded
>>> platform?
>>
>> You might look at XDR, which is meant for communication between
>> unlike devices. It may or may not do what you want, but it is
>> conveniently well standardized. (It is in a few internet RFCs.)
>
> Thank you for your suggestion. Anyway XDR seems to encode everything with a 4-bytes unit base. I have many small integer values and many bytes would be added without info.
>
> I'd like to use a more compact serialization format and simple to implement in embedded applications.
> BSON? JSON? Protocol Buffers? Other suggestions?
>

JSON has certain advantages. Since everything is text it is extensible
by interpretation. It's delimited by {} so boundaries are clear.
Interpreting fields is up to the protocol interpreter since the data
won't tell you its type, however it is reasonably easy to work out a
scheme of some sort.

e.g. {"int_array":[1,3,34,245],"string":"Hello, World","double":2.34}

It is certainly easy to come up with a more compact protocol and sane
updates to a specification are required in any case but it's easily
extensible and debugged and widely used.

On Thu, 20 Dec 2012 03:49:58 -0800, pozzugno wrote:

> Is there a better approach to exchange data on the wire between two C
> applications, considering the limitations of memory, the low speed and
> the length of messages of a small embedded platform?

Yes, there are better possibilities.
The easiest way to get a dense, binary, flexible protocol is to define
the protocol elements as Type/Length/Value triplets.
This means that each parameter in a request or response message consists
of a Type-code identifying the parameter, a Length byte stating the
length (in bytes) of the following data and the Value of the parameter.

For full compatibility between revisions of the protocol, you need to
observe a few rules:
- The Type-code field must have a fixed length, otherwise an
implementation can't know how to skip unsupported parameters. You should
leave plenty of room for adding new Type-codes (reserve at least one byte
more than you actually need for the current set of parameters).
- The Length field must have a fixed width, otherwise an implementation
doesn't know where the next parameter after an unsupported one starts.
For this reason, the Length field must always be encoded as bytes.
- The Type-code must identify the parameter type and its encoding (if
multiple incompatible encodings are possible). You may cheat a bit with
the Length field to omit leading 0-bytes on a field that encodes a single
integer (and thus later extend a byte to a two- or four-byte value), but
the documentation should read something like: "Type-code 0x0042: Value of
register T1; 4 bytes".
- Type-codes can never be reused or changed. Once they are assigned a
meaning, that meaning is set in stone. If you find later that it does not
fit anymore (e.g. data structure is extended), use a new Type-code. If
both old and new receivers must be supported, it might be best to use the
new code only for the additional data.

The parsers for these protocols work like this:
1. Read sizeof(Type-code) bytes as Type-code
2. Read sizeof(Length) bytes as Length
3. Read Length bytes as Value
4. Call the handler function that was registered for the Type-code (if
any)
5. Repeat steps 1 to 4 until end of the message
6. (Optional) Signal the end of the message to the handlers.

Bart v Ingen Schenau

Il 20/12/2012 15:09, Eric Sosman ha scritto:
>>>> At first this approach was good, but now I'm finding many problems.
>>>> Mostly I have to keep the two applications synchronized: if I add
>>>> a parameter (maybe in the middle of the answer, because an array
>>>> size is increased) or enlarge a numerical values (from 1 to 2 bytes)
>>>> or something else, I have to change both applications.
>>>> I usually want to have a retro-compatible master application
>>>> (master should communicate well with new or older slaves
>>>> application), so the code will be filled with many ifs
>>>> (if the slave answer version is 1.0 than there is one parameter
>>>> here, if it is 1.2 the array size is 3 and not 2...)
>>>
>>>> Is there a better approach to exchange data on the wire between
>>>> two C applications, considering the limitations of memory,
>>>> the low speed and the length of messages of a small embedded
>>>> platform?
>>>
>>> You might look at XDR, which is meant for communication between
>>> unlike devices. It may or may not do what you want, but it is
>>> conveniently well standardized. (It is in a few internet RFCs.)
>>
>> Thank you for your suggestion. Anyway XDR seems to encode everything
>> with a 4-bytes unit base. I have many small integer values and many
>> bytes would be added without info.
>>
>> I'd like to use a more compact serialization format and simple to
>> implement in embedded applications.
>> BSON? JSON? Protocol Buffers? Other suggestions?
>
> The fact that you're coding in C seems irrelevant to the
> question, which is all about the wire format. You might get
> better "outside the box" answers in a general programming
> forum than you will here.

Yes, I'll make this question also on comp.arch.embedded. I asked here,
because I was trying to find some read-to-use C libraries. If they
doesn't exist, I'll try to write my own.


> From your description, it looks like you have three concerns:
> flexibility, density, and simplicity. I doubt there's any one
> choice that gets close to all three corners of the triangle at
> the same time, so you'll probably have to settle for something
> that's a little bit rigid and/or bloated and/or complex -- and
> it's up to you to figure out what the trade-offs are.
>
> I've used protocol buffers (though not in C) and found them
> flexible and reasonably dense. I don't know whether implementations
> in C exist; if not, I think a full-fledged implementation might be
> something of a challenge (protobuffs aren't fully polymorphic, but
> they benefit a lot from some O-O capabilities). Still, perhaps
> you could study protobuffs and write yourself a subsetted version
> that would meet your needs without being too daunting.

It seems there are a couple of C implementations of protobuf, maybe
somewhat limited. Aside this porting problem, do you think protobuf
could be a good solution in my case?


> If it were up to me (it's not, of course), I think I'd go
> for XDR despite concerns about bloat. It can be pretty flexible,
> and since support libraries already exist it scores well on
> simplicity.

One thing I didn't understood (I'm sorry if I ask silly questions).
A XDR message is a sequence of parameters (integers, strings,
structures...). The receiver should know the exact sequence of
parameters generated by the transmitter. So, if a new parameter is
added in the middle of the message in a new version of transmitter can't
be correctly decoded by an old version of the receiver.

I think protobuf solve this problem with an ID associated to each
parameter.


> However, I'd make a point of segregating the
> encoding and decoding from the rest of the program, so I could
> change my mind later. Once the implementation settles down I'd
> imagine flexibility might become less important, and at that
> point I could switch to a purpose-built high-density protocol.
> But while things are still in flux, I don't think I'd worry so
> much about the format density unless it's really awful.
>

Il 20/12/2012 18:15, Greg Martin ha scritto:
>>>> At first this approach was good, but now I'm finding many problems.
>>>> Mostly I have to keep the two applications synchronized: if I add
>>>> a parameter (maybe in the middle of the answer, because an array
>>>> size is increased) or enlarge a numerical values (from 1 to 2 bytes)
>>>> or something else, I have to change both applications.
>>>> I usually want to have a retro-compatible master application
>>>> (master should communicate well with new or older slaves
>>>> application), so the code will be filled with many ifs
>>>> (if the slave answer version is 1.0 than there is one parameter
>>>> here, if it is 1.2 the array size is 3 and not 2...)
>>>
>>>> Is there a better approach to exchange data on the wire between
>>>> two C applications, considering the limitations of memory,
>>>> the low speed and the length of messages of a small embedded
>>>> platform?
>>>
>>> You might look at XDR, which is meant for communication between
>>> unlike devices. It may or may not do what you want, but it is
>>> conveniently well standardized. (It is in a few internet RFCs.)
>>
>> Thank you for your suggestion. Anyway XDR seems to encode everything
>> with a 4-bytes unit base. I have many small integer values and many
>> bytes would be added without info.
>>
>> I'd like to use a more compact serialization format and simple to
>> implement in embedded applications.
>> BSON? JSON? Protocol Buffers? Other suggestions?
>>
>
> JSON has certain advantages. Since everything is text it is extensible
> by interpretation. It's delimited by {} so boundaries are clear.
> Interpreting fields is up to the protocol interpreter since the data
> won't tell you its type, however it is reasonably easy to work out a
> scheme of some sort.
>
> e.g. {"int_array":[1,3,34,245],"string":"Hello, World","double":2.34}
>
> It is certainly easy to come up with a more compact protocol and sane
> updates to a specification are required in any case but it's easily
> extensible and debugged and widely used.

I can't use strings to encode everything: the message size should
increase too much for my embedded platform.

pozz wrote:
> Il 20/12/2012 18:15, Greg Martin ha scritto:
>>
>> JSON has certain advantages. Since everything is text it is extensible
>> by interpretation. It's delimited by {} so boundaries are clear.
>> Interpreting fields is up to the protocol interpreter since the data
>> won't tell you its type, however it is reasonably easy to work out a
>> scheme of some sort.
>>
>> e.g. {"int_array":[1,3,34,245],"string":"Hello, World","double":2.34}
>>
>> It is certainly easy to come up with a more compact protocol and sane
>> updates to a specification are required in any case but it's easily
>> extensible and debugged and widely used.
>
> I can't use strings to encode everything: the message size should
> increase too much for my embedded platform.

If you want the simplicity and flexibility of JSON but more compact
messages, look at BSON.

--
Ian Collins

On 12/20/2012 5:50 PM, pozz wrote:
>[...]
> It seems there are a couple of C implementations of protobuf, maybe
> somewhat limited. Aside this porting problem, do you think protobuf
> could be a good solution in my case?

"Yes," or at least "Worth considering." The original design
goal of protocol buffers was to address two of what I think are
your three main concerns: flexibility and information density.
Simplicity got sacrificed (to some extent, anyhow) in favor of
the other two, but if you can find an existing C implementation --
or if you can use a mixture of C and C++ in your application --
the implementation complexity may not be that big a barrier.

Still and all: It's your call, not ours.

--
Eric Sosman
d

Shao Miller wrote:
> On 12/20/2012 06:49, wrote:
>>
>> Is there a better approach to exchange data on the wire between two
>> C applications, considering the limitations of memory, the low speed
>> and the length of messages of a small embedded platform?
>
> You could use sentinel values (like strings use the null terminator)
> or send the array size before sending the array. - Shao

That is a lie. You don't want to rape (rape? what is rape? Ask Barack Obama,
cuz he said "rape is rape"). the ... but you go along. You don't want to
rape, but Society says, "take your turn".