Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C++ > Portability and marshalling integral data

Reply
Thread Tools

Portability and marshalling integral data

 
 
Brian
Guest
Posts: n/a
 
      11-24-2009

In the past there's been discussion about how using memcpy
to assist marshalling integral data should be avoided due to
portability reasons. From some very limited testing themore
portable approach that uses bit shifting to output/marshall
integral data is roughly 10 to 15% slower than using memcpy.

I'm wondering if it is possible to set things up so that the
more portable functions are used in a more limited way than
what I've seen suggested here. For example if a server
has both big endian and little endian (Intel) clients, would
it work to have the server send a "big endian" stream to
the big endian clients and a "little endian" stream to
Intel clients -- then both of those types of clients could
use memcpy to read/demarshall integral data types?

I believe it is possible to use memcpy without having
alignment problems. I know it is possible to show
examples of alignment problems using memcpy, but I
don't think those are relevant to marshalling of data
structures that a compiler has set up to it's liking.

What am I missing? Something to do with compiler
flags?


Brian Wood
http://webEbenezer.net
 
Reply With Quote
 
 
 
 
Brian
Guest
Posts: n/a
 
      11-24-2009
On Nov 24, 4:30*pm, Paavo Helde <(E-Mail Removed)> wrote:
> Brian <(E-Mail Removed)> wrote in news:2ff925d9-9d6f-4285-b0d2-
> (E-Mail Removed):


>
> > I'm wondering if it is possible to set things up so that the
> > more portable functions are used in a more limited way than
> > what I've seen suggested here. *For example if a server
> > has both big endian and little endian (Intel) clients, would
> > it work to have the server send a "big endian" stream to
> > the big endian clients and a "little endian" stream to
> > Intel clients -- then both of those types of clients could
> > use memcpy to read/demarshall integral data types?

>
> Are you seriously claiming that a data stream can be pumped over the
> network faster than the client can do some bitshifting?
>


No, I'm saying it seems like there's a way to eliminate the
need for bit shifting when reading data. It isn't clear to
me what (portability) is lost by doing it that way.


Brian Wood
http://webEbenezer.net
 
Reply With Quote
 
 
 
 
James Kanze
Guest
Posts: n/a
 
      11-25-2009
On Nov 24, 10:41 pm, Brian <(E-Mail Removed)> wrote:
> On Nov 24, 4:30 pm, Paavo Helde <(E-Mail Removed)> wrote:


> > Brian <(E-Mail Removed)> wrote in news:2ff925d9-9d6f-4285-b0d2-
> > (E-Mail Removed):


> > > I'm wondering if it is possible to set things up so that
> > > the more portable functions are used in a more limited way
> > > than what I've seen suggested here. For example if a
> > > server has both big endian and little endian (Intel)
> > > clients, would it work to have the server send a "big
> > > endian" stream to the big endian clients and a "little
> > > endian" stream to Intel clients -- then both of those
> > > types of clients could use memcpy to read/demarshall
> > > integral data types?


> > Are you seriously claiming that a data stream can be pumped
> > over the network faster than the client can do some
> > bitshifting?


> No, I'm saying it seems like there's a way to eliminate the
> need for bit shifting when reading data. It isn't clear to me
> what (portability) is lost by doing it that way.


It's possible for the server to systematically send and receive
data in the internal format of the client, yes. Provided it
knows this format (the client tells it). Does it buy you
anything? I'm not really sure.

The issue is particularly important with regards to floating
point; if the protocol specifies a format with a fixed maximum
precision (e.g. IEEE double), and the machines at both ends
both support higher precision, then information is lost
unnecessarily.

In this regard, you might be interested in the BER encoding of
RealType. In the case of BER, the goal is to not loose
precision unless necessary, so the rule is always to use the
sender's format, with the receiver interpreting it. In
practice, I've never really seen it fully implemented, because
handling multiple formats (even if only on the read side) is so
complicated. In your case, you seem to be only addressing byte
ordering---I've seen several protocols already which support two
different byte orderings, and it's not that complicated. But
IMHO, it still doesn't buy you that much.

--
James Kanze
 
Reply With Quote
 
James Kanze
Guest
Posts: n/a
 
      11-25-2009
On Nov 24, 11:14 pm, Paavo Helde <(E-Mail Removed)> wrote:
> Brian <(E-Mail Removed)> wrote in news:568c275d-6c21-4bc7-95e4-
> (E-Mail Removed):
> > On Nov 24, 4:30 pm, Paavo Helde <(E-Mail Removed)> wrote:
> >> Brian <(E-Mail Removed)> wrote in news:2ff925d9-9d6f-4285-b0d2-
> >> (E-Mail Removed):


> >> > I'm wondering if it is possible to set things up so that
> >> > the more portable functions are used in a more limited
> >> > way than what I've seen suggested here. For example if a
> >> > server has both big endian and little endian (Intel)
> >> > clients, would it work to have the server send a "big
> >> > endian" stream to the big endian clients and a "little
> >> > endian" stream to Intel clients -- then both of those
> >> > types of clients could use memcpy to read/demarshall
> >> > integral data types?


> >> Are you seriously claiming that a data stream can be pumped
> >> over the network faster than the client can do some
> >> bitshifting?


> > No, I'm saying it seems like there's a way to eliminate the
> > need for bit shifting when reading data. It isn't clear to
> > me what (portability) is lost by doing it that way.


> When adding a new client platform with 36 or 64 bit ints, or
> 32-bit ints in shuffled byte order, for example, one has to
> alter the server code for preparing the data for them. Also
> the client has to communicate its exact data model to the
> server, and the server has to understand it. This makes both
> the client and server more complicated.


> Of course, this can be done, but the goal has to be stated.
> The performance increase probably is not measurable (does not
> exceed the network speed fluctuations), so it does not qualify
> as a valid goal. There might be no performance increase at
> all if the server and client are using different data models
> as the conversion has to be done somewhere anyway.


There is one non-performance motivation which might come into
play: if you expect a lot of different people, some very naive,
writing client code. During the connection, you have them send
a certain number of values in an "int", and you interpret this
int to determine native format, and adapt (or reject the
connection because you don't understand the format). In this
case, by adapting to the client format, you can work even if the
authors' of the client code are relatively naive, and don't
really understand the issues. (On the other hand, most such
programmers are probably programming for a Windows platform, so
just adopting Intel format might be sufficient.)

--
James Kanze
 
Reply With Quote
 
dragan
Guest
Posts: n/a
 
      11-25-2009
Brian wrote:
> In the past there's been discussion about how using memcpy
> to assist marshalling integral data should be avoided due to
> portability reasons. From some very limited testing themore
> portable approach that uses bit shifting to output/marshall
> integral data is roughly 10 to 15% slower than using memcpy.
>
> I'm wondering if it is possible to set things up so that the
> more portable functions are used in a more limited way than
> what I've seen suggested here. For example if a server
> has both big endian and little endian (Intel) clients, would
> it work to have the server send a "big endian" stream to
> the big endian clients and a "little endian" stream to
> Intel clients -- then both of those types of clients could
> use memcpy to read/demarshall integral data types?
>
> I believe it is possible to use memcpy without having
> alignment problems. I know it is possible to show
> examples of alignment problems using memcpy, but I
> don't think those are relevant to marshalling of data
> structures that a compiler has set up to it's liking.
>
> What am I missing? Something to do with compiler
> flags?
>
>


What you are doing is specifying a protocol. IP, for example, specifies (the
wrong ) byte order for packet information and such, hence all those
htonl-like macros. If you say the spec is little endian, than whatever the
other end of the wire is doesn't matter: THEY have to convert the incoming
data to conform to your protocol specification. There is more than just
endianness to consider, of course. You have to maintain the spec with what
you send out by controlling it with whatever you have to, like compiler
pragmas to control alignment and padding.


 
Reply With Quote
 
James Kanze
Guest
Posts: n/a
 
      11-25-2009
On Nov 25, 11:45 am, "dragan" <(E-Mail Removed)> wrote:
> Brian wrote:
> > In the past there's been discussion about how using memcpy
> > to assist marshalling integral data should be avoided due to
> > portability reasons. From some very limited testing themore
> > portable approach that uses bit shifting to output/marshall
> > integral data is roughly 10 to 15% slower than using memcpy.


> > I'm wondering if it is possible to set things up so that the
> > more portable functions are used in a more limited way than
> > what I've seen suggested here. For example if a server has
> > both big endian and little endian (Intel) clients, would it
> > work to have the server send a "big endian" stream to the
> > big endian clients and a "little endian" stream to Intel
> > clients -- then both of those types of clients could use
> > memcpy to read/demarshall integral data types?


> > I believe it is possible to use memcpy without having
> > alignment problems. I know it is possible to show examples
> > of alignment problems using memcpy, but I don't think those
> > are relevant to marshalling of data structures that a
> > compiler has set up to it's liking.


> > What am I missing? Something to do with compiler flags?


> What you are doing is specifying a protocol. IP, for example,
> specifies (the wrong ) byte order for packet information
> and such, hence all those htonl-like macros.


There is no right or wrong (although I too prefer little
endian). IP is big endian because all of the backbones of the
network (where such protocols are developed) and the machines on
which large scale servers are run are all big endian. (Also,
htonl was an early hack, before all of the issues are
understood. I wouldn't expect to see it in modern code.)

> If you say the spec is little endian, than whatever the other
> end of the wire is doesn't matter: THEY have to convert the
> incoming data to conform to your protocol specification.


Well, usually you negociate with your clients, and find a
consensus. But yes, there is (or should be) a protocol
specification which specifies exactly how an integral value
should be represented (byte order, but also size and
representation). And everyone has to convert incoming data to
conform to it, and generate it correctly in outgoing data.

> There is more than just endianness to consider, of course. You
> have to maintain the spec with what you send out by
> controlling it with whatever you have to, like compiler
> pragmas to control alignment and padding.


You typically can't control it enough just by flipping a few
compiler switches.

--
James Kanze
 
Reply With Quote
 
Brian
Guest
Posts: n/a
 
      11-25-2009
On Nov 25, 4:10*am, James Kanze <(E-Mail Removed)> wrote:
> On Nov 24, 10:41 pm, Brian <(E-Mail Removed)> wrote:
>
>
> > No, I'm saying it seems like there's a way to eliminate the
> > need for bit shifting when reading data. *It isn't clear to me
> > what (portability) is lost by doing it that way.

>
> It's possible for the server to systematically send and receive
> data in the internal format of the client, yes. *Provided it
> knows this format (the client tells it). *Does it buy you
> anything? *I'm not really sure.
>


Since the task is handled by the server, the clients have an
easier time of it. Presumably the server developer wants
to have clients, so it seems to me a small step to make the
service more attractive. The expectation is for servers to
be built with ample hardware so this just continues that.
(I think the numbers I wrote in the original post are a
little high. After checking into this more I would say that
the bit shifting version is generally less than 11% slower
than the memcpy version for writing/output. It would be a
good idea to check what the difference is like for reading
as well, but for now I'm guessing it is similar.)


> The issue is particularly important with regards to floating
> point; if the protocol specifies a format with a fixed maximum
> precision (e.g. IEEE double), and the machines at both ends
> both support higher precision, then information is lost
> unnecessarily.


That's interesting. I'm working on getting things working
with integers first. After that I hope to work on floating
point issues. It may not work to have the same sort of
principle for floating point as for integers, but I don't
think that will affect the integer approach.


Brian Wood
http://webEbenezer.net
 
Reply With Quote
 
Brian
Guest
Posts: n/a
 
      11-25-2009
On Nov 25, 1:07*pm, Brian <(E-Mail Removed)> wrote:
> On Nov 25, 4:10*am, James Kanze <(E-Mail Removed)> wrote:
>
> > On Nov 24, 10:41 pm, Brian <(E-Mail Removed)> wrote:

>
> > > No, I'm saying it seems like there's a way to eliminate the
> > > need for bit shifting when reading data. *It isn't clear to me
> > > what (portability) is lost by doing it that way.

>
> > It's possible for the server to systematically send and receive
> > data in the internal format of the client, yes. *Provided it
> > knows this format (the client tells it). *Does it buy you
> > anything? *I'm not really sure.

>
> Since the task is handled by the server, the clients have an
> easier time of it. *Presumably the server developer wants
> to have clients, so it seems to me a small step to make the
> service more attractive. *The expectation is for servers to
> be built with ample hardware so this just continues that.


I guess what I'm considering wouldn't do that as much as
possible, since both clients and servers would be possibly
formatting the data with bit shifts when sending. Some of
it is just the simplicity of the implementation -- all
reads/input are handled the same way. That helps me to
think about it without going crazy.


Brian Wood
http://webEbenezer.net
 
Reply With Quote
 
dragan
Guest
Posts: n/a
 
      11-25-2009
James Kanze wrote:
> On Nov 25, 11:45 am, "dragan" <(E-Mail Removed)> wrote:
>> Brian wrote:
>>> In the past there's been discussion about how using memcpy
>>> to assist marshalling integral data should be avoided due to
>>> portability reasons. From some very limited testing themore
>>> portable approach that uses bit shifting to output/marshall
>>> integral data is roughly 10 to 15% slower than using memcpy.

>
>>> I'm wondering if it is possible to set things up so that the
>>> more portable functions are used in a more limited way than
>>> what I've seen suggested here. For example if a server has
>>> both big endian and little endian (Intel) clients, would it
>>> work to have the server send a "big endian" stream to the
>>> big endian clients and a "little endian" stream to Intel
>>> clients -- then both of those types of clients could use
>>> memcpy to read/demarshall integral data types?

>
>>> I believe it is possible to use memcpy without having
>>> alignment problems. I know it is possible to show examples
>>> of alignment problems using memcpy, but I don't think those
>>> are relevant to marshalling of data structures that a
>>> compiler has set up to it's liking.

>
>>> What am I missing? Something to do with compiler flags?

>
>> What you are doing is specifying a protocol. IP, for example,
>> specifies (the wrong ) byte order for packet information
>> and such, hence all those htonl-like macros.

>
> There is no right or wrong (although I too prefer little
> endian).


It seems that NOBODY here knows how to parse this->

>> There is more than just endianness to consider, of course. You
>> have to maintain the spec with what you send out by
>> controlling it with whatever you have to, like compiler
>> pragmas to control alignment and padding.

>
> You typically can't control it enough just by flipping a few
> compiler switches.


If you write the spec to assume the conventions of the platform the software
was written on, then you can. Flip a few switches and whatever comes out is
to spec. Call it "an implicit specification" then. Then, the high-level spec
is: little endian, integer sizes, no padding, all integer data on natural
boundaries. Anything else anyone wants to know will be specified later.
Of course, nothing need to be specified if developer B is using the same
platform (compiler, os, hardware).


 
Reply With Quote
 
James Kanze
Guest
Posts: n/a
 
      11-26-2009
On Nov 25, 11:09 pm, "dragan" <(E-Mail Removed)> wrote:
> James Kanze wrote:
> > On Nov 25, 11:45 am, "dragan" <(E-Mail Removed)> wrote:


[...]
> >> There is more than just endianness to consider, of course.
> >> You have to maintain the spec with what you send out by
> >> controlling it with whatever you have to, like compiler
> >> pragmas to control alignment and padding.


> > You typically can't control it enough just by flipping a few
> > compiler switches.


> If you write the spec to assume the conventions of the
> platform the software was written on, then you can.


If you write the spec to assume the conventions of the platform
you're working on, you may not have to flip any switches. But
of course, your code won't be portable; it won't implement the
same spec on a different platform.

Historically, this mistake was made a lot in the early days of
networking. XDR, for example, follows exactly the conventions
of the Motorola 68000, and until fairly recently, you could just
memory dump data from a 32 bit Sparc, and it would be conform
to the protocol. (I think more modern Sparc's require 8 byte
alignment of double, whereas the protocol only requires 4 byte
alignment.) Of course, if you're on an Intel platform, you'll
need some extra work. (This works against Brian's idea, since
most servers are still big-endian, where as Intel dominates
the client side overwhelmingly.)

> Flip a few switches and whatever comes out is to spec. Call it
> "an implicit specification" then. Then, the high-level spec
> is: little endian, integer sizes, no padding, all integer data
> on natural boundaries. Anything else anyone wants to know will
> be specified later.


2's complement as well, of course.

Lucky for us today, none of the people designing protocols were
on Unisys platforms. (In fact, most of the early work on IP
and TCP was done on DEC platforms. And the byte oriented DECs
are little endian. But I think when the work was being done, it
was still largely DEC 10's: word addressed, with 36 bit words,
and by convention, 5 seven bit bytes in each word, with one left
over bit. At least they didn't base the protocol on that.)

> Of course, nothing need to be specified if developer B is
> using the same platform (compiler, os, hardware).


Nothing needs to be specified if you can guarantee that for all
time, all future developers will be using the same platform
(compiler, including version, os, hardware). As soon as you
admit that some future version might be compiled with a later
version of the compiler, you're running a risk. (Note that the
byte order in a long changed from one version to the next in
Microsoft compilers for Intel. And the size of a long depends
on a compiler switch with most Unix systems.)

--
James Kanze
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Portable marshalling of floating point data Brian C++ 3 12-14-2009 09:51 AM
C/C++ language proposal: Change the 'case expression' from "integral constant-expression" to "integral expression" Adem C++ 42 11-04-2008 12:39 PM
C/C++ language proposal: Change the 'case expression' from "integral constant-expression" to "integral expression" Adem C Programming 45 11-04-2008 12:39 PM
Ruby on Rails session and Data Marshalling Summercool Ruby 1 09-11-2007 08:14 AM
Marshalling cyclic data structures Nathan Weston Ruby 5 11-03-2003 03:07 AM



Advertisments