«

Hello,

Does anyone know a (preferably open-source) multi-platform C or C++
library that would be able to write and read C/C++ doubles and floats
to/from streambuf, char array or similar device in IEEE 754 with
reasonably optimal precision and performance?

The purpose is to exchange serialized doubles and floats between C/C++
and Java programs where Java serialization rules are used.

I tried to search for it on the Internet but, surprisingly, could not
find any mature code for this seemingly common purpose.

Thanks in advance,
-Pavel

On Thu, 06 Mar 2008 04:01:51 GMT, Pavel
<dot_com_yahoo@paultolk_reverse.yourself> wrote in comp.lang.c++:

> Hello,

I'm having trouble understanding what you are asking for.

> Does anyone know a (preferably open-source) multi-platform C or C++
> library that would be able to write and read C/C++ doubles and floats
> to/from streambuf, char array or similar device in IEEE 754 with
> reasonably optimal precision and performance?

Assuming that your platform uses an IEEE format for doubles and
floats, about the fastest possible way to write them would be to a
binary file with fwrite().

> The purpose is to exchange serialized doubles and floats between C/C++
> and Java programs where Java serialization rules are used.

I have no idea what Java serialization of floating point types looks
like. Are you talking about writing to a text file or a binary file?
The IEEE 754 standard does not specify a text representation.

> I tried to search for it on the Internet but, surprisingly, could not
> find any mature code for this seemingly common purpose.

If you need a text file, fprintf() or stream insertion (<<) to a text
stream with appropriate format and precision specifiers.

If you are looking for some faster method of producing text
representations than the standard C or C++ library formatted output
functions, you might well find some, but they might not meet your
definition of "multi-platform", depending on how broadly you define
that term.

You could write, or find, code that takes advantage of the
implementation-defined details of the internal format that might
convert it to text faster than standard functions, but it would not be
directly portable across platforms with different endian
representations of floating point values.

--
Jack Klein
Home: http://JK-Technology.Com
FAQs for
comp.lang.c http://c-faq.com/
comp.lang.c++ http://www.parashift.com/c++-faq-lite/
alt.comp.lang.learn.c-c++
http://www.club.cc.cmu.edu/~ajo/docs/FAQ-acllc.html

On Mar 6, 7:30 am, Jack Klein <jackkl...@spamcop.net> wrote:
> On Thu, 06 Mar 2008 04:01:51 GMT, Pavel
> <dot_com_yahoo@paultolk_reverse.yourself> wrote in comp.lang.c++:

> I'm having trouble understanding what you are asking for.

> > Does anyone know a (preferably open-source) multi-platform C
> > or C++ library that would be able to write and read C/C++
> > doubles and floats to/from streambuf, char array or similar
> > device in IEEE 754 with reasonably optimal precision and
> > performance?

> Assuming that your platform uses an IEEE format for doubles
> and floats, about the fastest possible way to write them would
> be to a binary file with fwrite().

That would be true if it worked, but it doesn't.

Just saying that doubles and floats are IEEE isn't enough to
specify the format, and while both my Linux PC's and my Sparcs
use IEEE, the format is different on each. What *usually* works
(although formally undefined behavior) is to convert the address
of the double to a uint64_t* (or the address of the float to a
uint32_t*), and output that in the usual way, e.g.:

dest.put( value >> 56 ) ;
dest.put( value >> 48 ) ;
dest.put( value >> 40 ) ;
dest.put( value >> 32 ) ;
dest.put( value >> 24 ) ;
dest.put( value >> 16 ) ;
dest.put( value >> 8 ) ;
dest.put( value ) ;

To be totally generic, of course, you need to extract the
different parts of the double, and reassemble them. Something
like:

bool isNeg = source < 0 ;
if ( isNeg ) {
source = - source ;
}
int exp ;
if ( source == 0.0 ) {
exp = 0 ;
} else {
source = ldexp( frexp( source, &exp ), 53 ) ;
exp += 1022 ;
}
uint64_t mant = static_cast< uint64_t >( value ) ;
dest.put( (isNeg ? 0x80 : 0x00) | exp >> 4 ) ;
dest.put( ((exp << 4) & 0xF0) | ((mant >> 4 & 0x0F) ) ;
dest.put( mant >> 40 ) ;
dest.put( mant >> 32 ) ;
dest.put( mant >> 24 ) ;
dest.put( mant >> 16 ) ;
dest.put( mant >> 8 ) ;
dest.put( mant ) ;

(Interestingly enough, when I measured, I found that it wasn't
as slow as it looks. Note that it should work even if the host
doesn't use IEEE, e.g. an IBM or Unisys mainfram, but I've not
been able to actually test it there.)

> > The purpose is to exchange serialized doubles and floats
> > between C/C++ and Java programs where Java serialization
> > rules are used.

> I have no idea what Java serialization of floating point types
> looks like. Are you talking about writing to a text file or a
> binary file? The IEEE 754 standard does not specify a text
> representation.

Java uses a binary format: for floating point, it's basically
IEEE, output in big endian, the same as XDR, but (I think)
without the alignment requirements. (Formally, I don't think
that IEEE specifies where in the "word" the bits for the sign,
exponant and mantissa are placed, but practically, I've never
heard of an implementation where the order---from high bit to
low---wasn't sign, exponent, mantissa.)

--
James Kanze (GABI Software) email:
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

On 2008-03-05 23:01:51 -0500, Pavel
<dot_com_yahoo@paultolk_reverse.yourself> said:

>
> Does anyone know a (preferably open-source) multi-platform C or C++
> library that would be able to write and read C/C++ doubles and floats
> to/from streambuf, char array or similar device in IEEE 754 with
> reasonably optimal precision and performance?

http://netlib2.cs.utk.edu/fp/dtoa.c

The general idea is that you convert floating point values to a base 10
text representation with as many digits after the decimal point as are
needed to distinguish the value being written from the two neighboring
values. Sometimes you need very few ("2.0"), and occasioinally you need
quite a few. Caution: it's hackerhead C code, and requires a fair
amount of study to understand what it's doing. There's also a paper
somewhere (I've lost track of it) that gives the theoretical
underpinnings.

--
Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com) Author of "The
Standard C++ Library Extensions: a Tutorial and Reference
(www.petebecker.com/tr1book)

James Kanze wrote:
> On Mar 6, 7:30 am, Jack Klein <jackkl...@spamcop.net> wrote:
>> On Thu, 06 Mar 2008 04:01:51 GMT, Pavel
>> <dot_com_yahoo@paultolk_reverse.yourself> wrote in comp.lang.c++:
>
>> I'm having trouble understanding what you are asking for.
>
>>> Does anyone know a (preferably open-source) multi-platform C
>>> or C++ library that would be able to write and read C/C++
>>> doubles and floats to/from streambuf, char array or similar
>>> device in IEEE 754 with reasonably optimal precision and
>>> performance?
>
>> Assuming that your platform uses an IEEE format for doubles
>> and floats, about the fastest possible way to write them would
>> be to a binary file with fwrite().
>
> That would be true if it worked, but it doesn't.
>
> Just saying that doubles and floats are IEEE isn't enough to
> specify the format, and while both my Linux PC's and my Sparcs
> use IEEE, the format is different on each. What *usually* works
> (although formally undefined behavior) is to convert the address
> of the double to a uint64_t* (or the address of the float to a
> uint32_t*), and output that in the usual way, e.g.:
>
> dest.put( value >> 56 ) ;
> dest.put( value >> 48 ) ;
> dest.put( value >> 40 ) ;
> dest.put( value >> 32 ) ;
> dest.put( value >> 24 ) ;
> dest.put( value >> 16 ) ;
> dest.put( value >> 8 ) ;
> dest.put( value ) ;
>
> To be totally generic, of course, you need to extract the
> different parts of the double, and reassemble them. Something
> like:
>
> bool isNeg = source < 0 ;
> if ( isNeg ) {
> source = - source ;
> }
> int exp ;
> if ( source == 0.0 ) {
> exp = 0 ;
> } else {
> source = ldexp( frexp( source, &exp ), 53 ) ;
> exp += 1022 ;
> }
> uint64_t mant = static_cast< uint64_t >( value ) ;
> dest.put( (isNeg ? 0x80 : 0x00) | exp >> 4 ) ;
> dest.put( ((exp << 4) & 0xF0) | ((mant >> 4 & 0x0F) ) ;
> dest.put( mant >> 40 ) ;
> dest.put( mant >> 32 ) ;
> dest.put( mant >> 24 ) ;
> dest.put( mant >> 16 ) ;
> dest.put( mant >> 8 ) ;
> dest.put( mant ) ;
>
> (Interestingly enough, when I measured, I found that it wasn't
> as slow as it looks. Note that it should work even if the host
> doesn't use IEEE, e.g. an IBM or Unisys mainfram, but I've not
> been able to actually test it there.)
Thanks James, this is the closest to my problem I got so far.

I did not want to write myself and was looking for a library because I
am lazy and some corner cases should be worked out (native
representation does not necessarily represent all same set of numbers as
IEEE 754 representations). It's quite a bit of coding and testing so I
wanted to find if someone else has already done that before getting into
it. I guess I will start with this code of yours and then try to work my
way using standard C++. Certainly #ifdef - guarded multi-platform
version would do faster for each supported architecture but hopefully
the performance of the standard C++ code will suffice for my purpose.

Regards
-Pavel

Pete Becker wrote:
> On 2008-03-05 23:01:51 -0500, Pavel
> <dot_com_yahoo@paultolk_reverse.yourself> said:
>
>>
>> Does anyone know a (preferably open-source) multi-platform C or C++
>> library that would be able to write and read C/C++ doubles and floats
>> to/from streambuf, char array or similar device in IEEE 754 with
>> reasonably optimal precision and performance?
>
> http://netlib2.cs.utk.edu/fp/dtoa.c
>
> The general idea is that you convert floating point values to a base 10
> text representation with as many digits after the decimal point as are
> needed to distinguish the value being written from the two neighboring
> values. Sometimes you need very few ("2.0"), and occasioinally you need
> quite a few. Caution: it's hackerhead C code, and requires a fair amount
> of study to understand what it's doing. There's also a paper somewhere
> (I've lost track of it) that gives the theoretical underpinnings.
>
Thanks Pete -- that is useful piece of code you suggested, but in this
case I am not after decimal formatting / parsing but about the
conversion of native C++ float/doubles into most suitable single- &
double- precision IEEE 754 and back, basically some API along these
lines (assuming char is an 8-bit byte):

enum RetCode { OK, DOMAIN_ERROR };

RetCode doubleToIeee754Single(double, char *buf); // 4-byte buf
RetCode doubleToIeee754Double(double, char *buf); // 8-byte buf
RetCode floatToIeee754Single(float, char *buf); // 4-byte buf
RetCode floatToIeee754Double(float, char *buf); // 8-byte buf
RetCode singleIeee754ToDouble(const char *buf, double &); // 4-byte buf
RetCode doubleIeee754ToDouble(const char *buf, double &); // 8-byte buf
RetCode singleIeee754ToFloat(const char *buf, float &); // 4-byte buf
RetCode doubleIeee754ToFloat(const char *buf, float &); // 8-byte buf

with all NaNs, infinities etc somehow reasonably worked out, for several
platforms (possibly, on a per-platform basis).

Java does it somehow so now I wonder if I can legally scavenge some
open-source JVM code.. will see what I can find or write.

-Pavel

On Mar 7, 6:17 am, Pavel <dot_com_yahoo@paultolk_reverse.yourself>
wrote:
> James Kanze wrote:

> I did not want to write myself and was looking for a library
> because I am lazy and some corner cases should be worked out
> (native representation does not necessarily represent all same
> set of numbers as IEEE 754 representations).

If you have to deal with a native representation that is not
IEEE, then you have to define behavior for corner cases. An IBM
double, for example, has more precision (but less range) than an
IEEE double: I suppose when outputting you round (the code I
posted truncates!), but what happens when you input something
that it too large.

I've heard that there may be similar cases even between IEEE
representations, at least where NaN's are involved---a non
trapping NaN may trap elsewhere, or something like that.

> It's quite a bit of coding and testing so I wanted to find if
> someone else has already done that before getting into it. I
> guess I will start with this code of yours and then try to
> work my way using standard C++. Certainly #ifdef - guarded
> multi-platform version would do faster for each supported
> architecture but hopefully the performance of the standard C++
> code will suffice for my purpose.

Well, I'd very definitely not invest the effort in supporting
machines without IEEE until I had to. (The three I know of
today are all mainframes: one uses base 16, one uses base 8, and
the one that uses base 2 has 72 bit doubles. On the latter two,
you'll need special handling for ints as well, because they
don't use 2's complement either: there's one with 48 bit signed
magnitude, with 8 bits required to be 0, and the other uses 36
bit 1's complement. Porting networked software to those should
be quite amusing---except that both have Intel based front-ends
to handle the networking.)

--
James Kanze (GABI Software) email:
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

On Mar 5, 11:01 pm, Pavel <dot_com_yahoo@paultolk_reverse.yourself>
wrote:
> Hello,
>
> Does anyone know a (preferably open-source) multi-platform C or C++
> library that would be able to write and read C/C++ doubles and floats
> to/from streambuf, char array or similar device in IEEE 754 with
> reasonably optimal precision and performance?
>
> The purpose is to exchange serialized doubles and floats between C/C++
> and Java programs where Java serialization rules are used.
>
> I tried to search for it on the Internet but, surprisingly, could not
> find any mature code for this seemingly common purpose.

Would this help ?
http://tpl.sourceforge.net/

Its an open source serialization library written in C.

James Kanze wrote:
> On Mar 7, 6:17 am, Pavel <dot_com_yahoo@paultolk_reverse.yourself>
> wrote:
>> James Kanze wrote:
>
>> I did not want to write myself and was looking for a library
>> because I am lazy and some corner cases should be worked out
>> (native representation does not necessarily represent all same
>> set of numbers as IEEE 754 representations).
>
> If you have to deal with a native representation that is not
> IEEE, then you have to define behavior for corner cases.
True, that's why I hoped someone did it before me in a reasonable way .

> An IBM
> double, for example, has more precision (but less range) than an
> IEEE double: I suppose when outputting you round (the code I
> posted truncates!), but what happens when you input something
> that it too large.
On input will have to check whether IEEE representation exceeds
implementation-specific limits from <climit> and <cfloat>; if it does,
throw domain_error or do something similar; also, for output need to
check whether the number to be written is within the respective (single-
or double-) IEEE limit. Long story short, lots of work to do..

> I've heard that there may be similar cases even between IEEE
> representations, at least where NaN's are involved---a non
> trapping NaN may trap elsewhere, or something like that.
True

> Well, I'd very definitely not invest the effort in supporting
> machines without IEEE until I had to. (The three I know of
> today are all mainframes: one uses base 16, one uses base 8, and
> the one that uses base 2 has 72 bit doubles. On the latter two,
> you'll need special handling for ints as well, because they
> don't use 2's complement either: there's one with 48 bit signed
> magnitude, with 8 bits required to be 0, and the other uses 36
> bit 1's complement. Porting networked software to those should
> be quite amusing---except that both have Intel based front-ends
> to handle the networking.)
You are right about those -- I do not need them (not right now, anyway);
the main problem even with mostly IEEE-ish architecture that you still
have to detect it (even if only to make sure the preprocessor balks on
unsupported architectures) and for that you need to detect the compiler
first -- both make and version -- because the macros that define the
architectures are implementation-specific and sometimes change between
versions -- and for that you have to first make a decision what versions
of compilers to support. I will need to support 4 compiler makes at
least; as I said, lots of work, at least 90% of which was already done
by others multiple times, better than I ever will.

Anyway, time to stop whining and start coding. Wish me luck.

-Pavel

On 8 mar, 05:53, Pavel <dot_com_yahoo@paultolk_reverse.yourself>
wrote:
> James Kanze wrote:
> > On Mar 7, 6:17 am, Pavel <dot_com_yahoo@paultolk_reverse.yourself>
> > wrote:
> >> James Kanze wrote:

> >> I did not want to write myself and was looking for a library
> >> because I am lazy and some corner cases should be worked out
> >> (native representation does not necessarily represent all same
> >> set of numbers as IEEE 754 representations).

> > If you have to deal with a native representation that is not
> > IEEE, then you have to define behavior for corner cases.

> True, that's why I hoped someone did it before me in a
> reasonable way .

The problem isn't just the doing. The problem is specifying
what you mean by "a reasonable way".

> > An IBM
> > double, for example, has more precision (but less range) than an
> > IEEE double: I suppose when outputting you round (the code I
> > posted truncates!), but what happens when you input something
> > that it too large.

> On input will have to check whether IEEE representation
> exceeds implementation-specific limits from <climit> and
> <cfloat>; if it does, throw domain_error or do something
> similar; also, for output need to check whether the number to
> be written is within the respective (single- or double-) IEEE
> limit. Long story short, lots of work to do..

If the absolute value you read is too big for your internal
representation, it's obviously an error (setting failbit would
be the usual behavior). If the absolute value is smaller that
min(), but not zero, however: is it an error, or do you use 0.0?
What if the value you read has more precision than you can
represent? Is it an error, or do you round.

--
James Kanze (GABI Software) email:
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34