Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   C++ (http://www.velocityreviews.com/forums/f39-c.html)
-   -   Binary data representation (http://www.velocityreviews.com/forums/t281128-binary-data-representation.html)

Charles T. 02-04-2004 03:46 PM

Binary data representation
 
Hi,

I currently writing a serialize/unserialize architecture. The read/write
function will read/write from a binary file.

My question is is there some sort on defined standart to use when
representing data type (int , int32, int64, double, string, etc....) ?


Thanks,





Phlip 02-04-2004 04:05 PM

Re: Binary data representation
 
Charles T. wrote:

> I currently writing a serialize/unserialize architecture. The read/write
> function will read/write from a binary file.


Why a binary file?

> My question is is there some sort on defined standart to use when
> representing data type (int , int32, int64, double, string, etc....) ?


No. There is not even a "standard" for what order bytes go inside an int.

The least heinous data format is XML. You can write very simple or very
complex data structures in it, and you can read those structures in a text
editor.

But XML can be a little obese. Some data formats are compressed XML.

--
Phlip
http://www.xpsd.org/cgi-bin/wiki?Tes...UserInterfaces



AirPete 02-04-2004 04:36 PM

Re: Binary data representation
 
[snip]

> The least heinous data format is XML. You can write very simple or
> very complex data structures in it, and you can read those structures
> in a text editor.
>
> But XML can be a little obese. Some data formats are compressed XML.


I would reccomend this, also.
The game Age of Mythology uses XML compressed with zLib compatible
compression, and it generates very compact but easily decoded files.
You can get zLib here:
http://www.zlib.org

- Pete



Thomas Matthews 02-04-2004 05:07 PM

Re: Binary data representation
 
Charles T. wrote:

> Hi,
>
> I currently writing a serialize/unserialize architecture. The read/write
> function will read/write from a binary file.


There has been much discussion on Serialization and Persistence in
this newsgroup and news:comp.lang.c. Use a search engine and look
for some ideas.


> My question is is there some sort on defined standart to use when
> representing data type (int , int32, int64, double, string, etc....) ?


There is no standard, from platform to platform. On some platforms,
there may be no standards between OS versions or compiler versions.
For better portability, write out the data in a consistent form
(i.e. uint64 == 64 bits, little endian) and let the programs convert
the data into the native representation.

Remember, when serializing, that the size of a structure may not
be the sum of the size of its members. Compilers are allowed to
add "padding bytes" between members.

Pointers don't store well. There is a very small probability
that an OS will allocate a variable in the same place for each
execution of a program.

Since pointers don't store well, don't store strings as pointers.
Store text as <quantity, text> or <text, sentinel character>.

See section [35] of the C++ FAQ (about serialization):
http://www.parashift.com/c++-faq-lit...alization.html

>
>
> Thanks,
>



--
Thomas Matthews

C++ newsgroup welcome message:
http://www.slack.net/~shiva/welcome.txt
C++ Faq: http://www.parashift.com/c++-faq-lite
C Faq: http://www.eskimo.com/~scs/c-faq/top.html
alt.comp.lang.learn.c-c++ faq:
http://www.raos.demon.uk/acllc-c++/faq.html
Other sites:
http://www.josuttis.com -- C++ STL Library book


Gianni Mariani 02-04-2004 05:21 PM

Re: Binary data representation
 
Phlip wrote:
> Charles T. wrote:
>
>
>>I currently writing a serialize/unserialize architecture. The read/write
>>function will read/write from a binary file.

>
>
> Why a binary file?
>
>
>>My question is is there some sort on defined standart to use when
>>representing data type (int , int32, int64, double, string, etc....) ?

>
>
> No. There is not even a "standard" for what order bytes go inside an int.
>
> The least heinous data format is XML. You can write very simple or very
> complex data structures in it, and you can read those structures in a text
> editor.
>
> But XML can be a little obese. Some data formats are compressed XML.


If you're talking about a real-time (streaming) system, the XML overhead
may be too much of a price to pay.

In 1999 I built a binary XML format that could be "parsed" in a fraction
of the time. But for some systems, even this one was too expensive.


AirPete 02-04-2004 05:36 PM

Re: Binary data representation
 
Gianni Mariani wrote:
[snip]
>
> In 1999 I built a binary XML format that could be "parsed" in a
> fraction of the time. But for some systems, even this one was too
> expensive.


Would you mind posting your implementation? I would be interested in seeing
it.
Thanks!

- Pete



Geoff Macartney 02-04-2004 10:04 PM

Re: Binary data representation
 
You might want to look (depending on your application area and on
whether you have time to learn it) at ASN.1, which is an ITU standard to
provide "a notation for defining data structures [and] a defined
(machine-independent) encoding for those data structures".

Have a glance at www-sop.inria.fr/rodeo/personnel/hoschka/asn1.html,
www.asn1.org, or google will bring back lots of links.

Geoff Macartney

Charles T. wrote:

> Hi,
>
> I currently writing a serialize/unserialize architecture. The read/write
> function will read/write from a binary file.
>
> My question is is there some sort on defined standart to use when
> representing data type (int , int32, int64, double, string, etc....) ?
>
>
> Thanks,
>
>
>
>



Martijn Lievaart 02-04-2004 10:22 PM

Re: Binary data representation
 
On Wed, 04 Feb 2004 12:21:16 -0500, Gianni Mariani wrote:

> In 1999 I built a binary XML format that could be "parsed" in a fraction
> of the time. But for some systems, even this one was too expensive.


No need to reinvent the wheel, have a look at ASN.1. Parsers abundand BTW.

M4


Gianni Mariani 02-05-2004 12:50 AM

Re: Binary data representation
 
Martijn Lievaart wrote:
> On Wed, 04 Feb 2004 12:21:16 -0500, Gianni Mariani wrote:
>
>
>>In 1999 I built a binary XML format that could be "parsed" in a fraction
>>of the time. But for some systems, even this one was too expensive.

>
>
> No need to reinvent the wheel, have a look at ASN.1. Parsers abundand BTW.


ASN.1 is different - the binary format I'm talking about has a 1:1
correlation to XML. The format was simply more efficient to parse than
XML text - admitedly the XML parser I wrote was slower than molasses in
a blizzard ... :-)


David Rasmussen 02-05-2004 02:36 AM

Re: Binary data representation
 
Charles T. wrote:
> Hi,
>
> I currently writing a serialize/unserialize architecture. The read/write
> function will read/write from a binary file.
>
> My question is is there some sort on defined standart to use when
> representing data type (int , int32, int64, double, string, etc....) ?
>


I have an application in which the compactness of binary representation
(as compared with, say, XML) is important, but where portability of that
binary file, regardless of endianess, is also important. My solution is
very simple: I just choose an endianess and stick with it, and make sure
to write/read one byte at a time to construct/reconstruct the data. It
works fine. The binary file is as compact as if I didn't care about
portability, and it works with all kinds of endianess. The reading and
the writing in principle takes a little longer because of the
disassembling/assembling that takes place here, but in practice it is
not a problem at all because of buffering. I just read, say, 1k at a
time and the problem disappears. Also, there are usually layers of
buffering involved anyway, in the OS, in the disk etc.

/David


All times are GMT. The time now is 06:02 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.