Optimize a serial protocol for data exchange between two C applications
 

I have two C applications running on embedded platforms. Actually they communicate through a serial connection (38400bps) with a proprietary binary protocol.

The master application sends commands to the slave application requesting its status or changing some settings. The answer to the status request is a sequence of binary data: numerical values in 1 or 2 or 4 bytes (in some cases, organized in arrays), small null-terminated strings, bitmask and so on.

At first this approach was good, but now I'm finding many problems. Mostly I have to keep the two applications synchronized: if I add a parameter (maybe in the middle of the answer, because an array size is increased) or enlarge a numerical values (from 1 to 2 bytes) or something else, I have to change both applications. I usually want to have a retro-compatible master application (master should communicate well with new or older slaves application), so the code will be filled with many ifs (if the slave answer version is 1.0 than there is one parameter here, if it is 1.2 the array size is 3 and not 2...)

Is there a better approach to exchange data on the wire between two C applications, considering the limitations of memory, the low speed and the length of messages of a small embedded platform?

Re: Optimize a serial protocol for data exchange between two C applications
 

On 12/20/2012 06:49, pozzugno@gmail.com wrote:
>
> Is there a better approach to exchange data on the wire between two C applications, considering the limitations of memory, the low speed and the length of messages of a small embedded platform?
>

You could use sentinel values (like strings use the null terminator) or
send the array size before sending the array. - Shao

Re: Optimize a serial protocol for data exchange between two C applications
 

pozzugno@gmail.com wrote:
(snip)

> At first this approach was good, but now I'm finding many problems.
> Mostly I have to keep the two applications synchronized: if I add
> a parameter (maybe in the middle of the answer, because an array
> size is increased) or enlarge a numerical values (from 1 to 2 bytes)
> or something else, I have to change both applications.
> I usually want to have a retro-compatible master application
> (master should communicate well with new or older slaves
> application), so the code will be filled with many ifs
> (if the slave answer version is 1.0 than there is one parameter
> here, if it is 1.2 the array size is 3 and not 2...)

> Is there a better approach to exchange data on the wire between
> two C applications, considering the limitations of memory,
> the low speed and the length of messages of a small embedded
> platform?

You might look at XDR, which is meant for communication between
unlike devices. It may or may not do what you want, but it is
conveniently well standardized. (It is in a few internet RFCs.)

-- glen

Re: Optimize a serial protocol for data exchange between two C applications
 

On Thursday, December 20, 2012 2:04:17 PM UTC+1, glen herrmannsfeldt wrote:
>
> > At first this approach was good, but now I'm finding many problems.
> > Mostly I have to keep the two applications synchronized: if I add
> > a parameter (maybe in the middle of the answer, because an array
> > size is increased) or enlarge a numerical values (from 1 to 2 bytes)
> > or something else, I have to change both applications.
> > I usually want to have a retro-compatible master application
> > (master should communicate well with new or older slaves
> > application), so the code will be filled with many ifs
> > (if the slave answer version is 1.0 than there is one parameter
> > here, if it is 1.2 the array size is 3 and not 2...)
>
> > Is there a better approach to exchange data on the wire between
> > two C applications, considering the limitations of memory,
> > the low speed and the length of messages of a small embedded
> > platform?
>
> You might look at XDR, which is meant for communication between
> unlike devices. It may or may not do what you want, but it is
> conveniently well standardized. (It is in a few internet RFCs.)

Thank you for your suggestion. Anyway XDR seems to encode everything with a 4-bytes unit base. I have many small integer values and many bytes would be added without info.

I'd like to use a more compact serialization format and simple to implement in embedded applications.
BSON? JSON? Protocol Buffers? Other suggestions?

Re: Optimize a serial protocol for data exchange between two C applications
 

pozzugno@gmail.com wrote:
> I have two C applications running on embedded platforms. Actually
> they communicate through a serial connection (38400bps) with a
> proprietary binary protocol.
>


Binary protocols are in general a form of premature optimization. But
we've all done it... it's not really a protocol; it's an interpreter.


> The master application sends commands to the slave application
> requesting its status or changing some settings. The answer to the
> status request is a sequence of binary data: numerical values in 1 or
> 2 or 4 bytes (in some cases, organized in arrays), small
> null-terminated strings, bitmask and so on.
>
> At first this approach was good, but now I'm finding many problems.
> Mostly I have to keep the two applications synchronized: if I add a
> parameter (maybe in the middle of the answer, because an array size
> is increased) or enlarge a numerical values (from 1 to 2 bytes) or
> something else, I have to change both applications. I usually want to
> have a retro-compatible master application (master should communicate
> well with new or older slaves application), so the code will be
> filled with many ifs (if the slave answer version is 1.0 than there
> is one parameter here, if it is 1.2 the array size is 3 and not
> 2...)
>
> Is there a better approach to exchange data on the wire between two C
> applications, considering the limitations of memory, the low speed
> and the length of messages of a small embedded platform?
>


As to having multiple protocol versions:

- Have only new commands added. V1.2 of the protocol is a
proper superset of V1.1

- Use dispatch through callback tables to support different "shapes"
within the protocol.

- try to manage the protocol as a stack, as a separately compiled and
"libraried" unit with its own regression suite. The regression suite
might be able to spawn a server and a client who both use a shared
memory to emulate a serial port. This allows it to run on a
workstation, which might reduce the cost of maintenance.

- use a scripting language to build a test driver for each version,
in each role.


--
Les Cargill

Re: Optimize a serial protocol for data exchange between two C applications
 

<pozzugno@gmail.com> wrote in message
news:5b183228-2ce7-4a8b-b321-058d455fea7b@googlegroups.com...
> I have two C applications running on embedded platforms. Actually they
> communicate through a serial connection (38400bps) with a proprietary
> binary protocol.
>
> The master application sends commands to the slave application requesting
> its status or changing some settings. The answer to the status request is
> a sequence of binary data: numerical values in 1 or 2 or 4 bytes (in some
> cases, organized in arrays), small null-terminated strings, bitmask and so
> on.
>
> At first this approach was good, but now I'm finding many problems. Mostly
> I have to keep the two applications synchronized: if I add a parameter
> (maybe in the middle of the answer, because an array size is increased) or
> enlarge a numerical values (from 1 to 2 bytes) or something else, I have
> to change both applications. I usually want to have a retro-compatible
> master application (master should communicate well with new or older
> slaves application), so the code will be filled with many ifs (if the
> slave answer version is 1.0 than there is one parameter here, if it is 1.2
> the array size is 3 and not 2...)

So at present the format of the returned data is hard-coded? That is, the
number of items, and the format of each item (1, 2 or 4 bytes etc).

The first problem then is the format of each item. In this case just tag
each value: preceded it with a byte indicating whether it is 1, 2 or 4 bytes
(sometimes this can be done without needing a separate byte) and perhaps
whether it's a string etc.

This could work if the reader knows what sequence to expect, and perhaps the
type, if not the exact format, of each item.

But if the type of an item can change, or you add a new one in the middle,
or even at the end, then the reader might be able to receive it, but it will
no longer understand what the items mean!

Then you will need to add information about the meaning of each item (an
identifier) and it gets more complicated. You mentioned serialisation and
JSON. But it's difficult to give advice without knowing more details or the
limitations you have (speed-wise for example). (Personally I wouldn't use
3rd party solutions; they tend to have a big learning curve, add more
dependencies, and usually completely dwarf my entire application. But that
might be just me...)

--
Bartc

Re: Optimize a serial protocol for data exchange between two C applications
 

On 12/20/2012 8:23 AM, pozzugno@gmail.com wrote:
> On Thursday, December 20, 2012 2:04:17 PM UTC+1, glen herrmannsfeldt wrote:
>>
>>> At first this approach was good, but now I'm finding many problems.
>>> Mostly I have to keep the two applications synchronized: if I add
>>> a parameter (maybe in the middle of the answer, because an array
>>> size is increased) or enlarge a numerical values (from 1 to 2 bytes)
>>> or something else, I have to change both applications.
>>> I usually want to have a retro-compatible master application
>>> (master should communicate well with new or older slaves
>>> application), so the code will be filled with many ifs
>>> (if the slave answer version is 1.0 than there is one parameter
>>> here, if it is 1.2 the array size is 3 and not 2...)
>>
>>> Is there a better approach to exchange data on the wire between
>>> two C applications, considering the limitations of memory,
>>> the low speed and the length of messages of a small embedded
>>> platform?
>>
>> You might look at XDR, which is meant for communication between
>> unlike devices. It may or may not do what you want, but it is
>> conveniently well standardized. (It is in a few internet RFCs.)
>
> Thank you for your suggestion. Anyway XDR seems to encode everything with a 4-bytes unit base. I have many small integer values and many bytes would be added without info.
>
> I'd like to use a more compact serialization format and simple to implement in embedded applications.
> BSON? JSON? Protocol Buffers? Other suggestions?

The fact that you're coding in C seems irrelevant to the
question, which is all about the wire format. You might get
better "outside the box" answers in a general programming
forum than you will here.

From your description, it looks like you have three concerns:
flexibility, density, and simplicity. I doubt there's any one
choice that gets close to all three corners of the triangle at
the same time, so you'll probably have to settle for something
that's a little bit rigid and/or bloated and/or complex -- and
it's up to you to figure out what the trade-offs are.

I've used protocol buffers (though not in C) and found them
flexible and reasonably dense. I don't know whether implementations
in C exist; if not, I think a full-fledged implementation might be
something of a challenge (protobuffs aren't fully polymorphic, but
they benefit a lot from some O-O capabilities). Still, perhaps
you could study protobuffs and write yourself a subsetted version
that would meet your needs without being too daunting.

If it were up to me (it's not, of course), I think I'd go
for XDR despite concerns about bloat. It can be pretty flexible,
and since support libraries already exist it scores well on
simplicity. However, I'd make a point of segregating the
encoding and decoding from the rest of the program, so I could
change my mind later. Once the implementation settles down I'd
imagine flexibility might become less important, and at that
point I could switch to a purpose-built high-density protocol.
But while things are still in flux, I don't think I'd worry so
much about the format density unless it's really awful.

--
Eric Sosman
esosman@comcast-dot-net.invalid

Re: Optimize a serial protocol for data exchange between two C applications
 

pozzugno@gmail.com wrote:

(snip, I wrote)
>> You might look at XDR, which is meant for communication between
>> unlike devices. It may or may not do what you want, but it is
>> conveniently well standardized. (It is in a few internet RFCs.)

> Thank you for your suggestion. Anyway XDR seems to encode everything
> with a 4-bytes unit base. I have many small integer values and many
> bytes would be added without info.

I haven't looked at it so recently, but that sounds right for
scalars. For arrays, I thought it packed them more. I am not
sure at all by now for structs.

-- glen

Re: Optimize a serial protocol for data exchange between two C applications
 

pozzugno wrote:
> I have two C applications running on embedded platforms. Actually
> they communicate through a serial connection (38400bps) with a
> proprietary binary protocol.

You could try posting in comp.arch.embedded, they might provide
some insight.

Re: Optimize a serial protocol for data exchange between two C applications
 

On 12-12-20 06:28 AM, glen herrmannsfeldt wrote:
> pozzugno@gmail.com wrote:
>
> (snip, I wrote)
>>> You might look at XDR, which is meant for communication between
>>> unlike devices. It may or may not do what you want, but it is
>>> conveniently well standardized. (It is in a few internet RFCs.)
>
>> Thank you for your suggestion. Anyway XDR seems to encode everything
>> with a 4-bytes unit base. I have many small integer values and many
>> bytes would be added without info.
>
> I haven't looked at it so recently, but that sounds right for
> scalars. For arrays, I thought it packed them more. I am not
> sure at all by now for structs.
>
> -- glen
>

AFAIK each member of an array is 4 bytes. Strings however are encoded
with single bytes preceded by a 4 byte length field and padded to a 4
byte boundary by 0 value bytes.