Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C++ > converting char to float (reading binary data from file)

Reply
Thread Tools

converting char to float (reading binary data from file)

 
 
coal@mailvault.com
Guest
Posts: n/a
 
      05-26-2008
On May 22, 1:58*am, James Kanze <(E-Mail Removed)> wrote:
> On May 21, 10:49 pm, Michael DOUBEZ <(E-Mail Removed)> wrote:
>
>
>
>
>
> > James Kanze a écrit :
> > > On 21 mai, 11:29, Michael DOUBEZ <(E-Mail Removed)> wrote:
> > >> itdevries a écrit :
> > >>> I'm trying to convert some char data I read from a binary
> > >>> file (using ifstream) to a float type. I've managed to
> > >>> convert the int types but now I need to do the float types
> > >>> as well but it doesn't seem to work. *The code below is what
> > >>> I'm trying to use. Anyone see any obvious errors? or have
> > >>> any hints/pointers?
> > >> I suppose charBuf is of the type char[] ? In this case
> > >> 'charBuf[ii] << offset' is 0 as soon as offset>=8 so you get
> > >> only 0s.
> > > Since when? *That's not what my compiler does, and it's not
> > > what the standard requires. *In expressions, char's are
> > > promoted to int's.

> > Then, it is my mistake and I am the worse for not checking. *I
> > am always careful to avoid writing code like that.
> > >> What's wrong with
> > >> memcpy(&floatRead,charBuf+startInd,sizeof(floatRea d));
> > >> ?
> > > Maybe the fact that it doesn't have the correct semantics? *(It
> > > might work, sometimes, but it's certainly a risky procedure.

> > More risky than using bitwise operators to build a float ?

>
> Yes.
>
> Basically, using the bitwise operators depends on the external
> representation being the same as the internal, i.e. they are
> both IEEE (or both something else, but I've never seen that
> case). *In practice, a lot of modern protocols do use IEEE, and
> most modern small and medium sized computers also use it
> internally, so if you are only targetting Windows and mainstream
> Unix, you're safe (and a few judicious #if on the values defined
> in <climits>, controlling an #error, can assure that it probably
> won't compile on systems which don't use IEEE).
>
> It's ugly, but the alternative is a lot more complex.
> (Interestingly, it's not that much slower on my machine. *Which
> rather surprised me.)
>
> > Concerning the semantic, I don't see what could be a correct
> > one in this case since reading a binary flow to build a float
> > is however heretical; unless you have some kind of standard
> > transform and assuming the capacities of representation are
> > equivalent across addressed architectures.

>
> As mentionned above, a lot of protocols (BER being the only
> exception which comes to mind) use IEEE format for floating
> point, as do a lot of machines. *In this case, reading the value
> as if it were the same sized unsigned integer type (uint32_t or
> uint64_t, depending on whether you're reading float or double),
> followed by some sort of nasty cast, will work.
>
> > What is uncorrect with dumping the memory area representing a
> > POD and reading it back in the same program ?

>
> The fact that there are still significant differences in the
> binary representation, particularly with regards to byte order.
> If you want to be really correctly, you need something like:
>
> * * uint32_t * * * * * *tmp ;
> * * operator>>( tmp ) ;
> * * if ( *this ) {
> * * * * float * * * * * * * f = 0.0 ;
> * * * * if ( (tmp & 0x7FFFFFFF) != 0 ) {
> * * * * * * f = ldexp( ((tmp & 0x007FFFFF) | 0x00800000),
> * * * * * * * * * * * *(int)((tmp & 0x7F800000) >> 23) - 126 - 24 ) ;
> * * * * }
> * * * * if ( (tmp & 0x80000000) != 0 ) {
> * * * * * * f = -f ;
> * * * * }
> * * * * dest = f ;
> * * }
> * * return *this ;
>
> You still read the type into the unsigned int (using shifts),
> but then you extract the fields as they are specified in the
> format, and use them to create a float, using standard functions
> to exploit their values in a portable fashion. *As I've said
> previously, this is probably more work than necessary, and until
> needed, I'd just go with the ugly cast (and the #if/#error, to
> ensure that it doesn't get used when not appropriate).
>




In Boost 1.35 they've added an optimization to take advantage of
contiguous collections of primitive data types. Here is a copy
of a file that is involved:

#ifndef BOOST_ARCHIVE_BASIC_BINARY_OPRIMITIVE_HPP
#define BOOST_ARCHIVE_BASIC_BINARY_OPRIMITIVE_HPP

// MS compatible compilers support #pragma once
#if defined(_MSC_VER) && (_MSC_VER >= 1020)
# pragma once
#endif

/////////
1/////////2/////////3/////////4/////////5/////////6/////////7/////////8
// basic_binary_oprimitive.hpp

// (C) Copyright 2002 Robert Ramey - http://www.rrsd.com .
// Use, modification and distribution is subject to the Boost Software
// License, Version 1.0. (See accompanying file LICENSE_1_0.txt or
copy at
// http://www.boost.org/LICENSE_1_0.txt)

// See http://www.boost.org for updates, documentation, and revision
history.

// archives stored as native binary - this should be the fastest way
// to archive the state of a group of obects. It makes no attempt to
// convert to any canonical form.

// IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
// ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON

#include <iosfwd>
#include <cassert>
#include <locale>
#include <cstddef> // size_t
#include <streambuf> // basic_streambuf
#include <string>

#include <boost/config.hpp>
#if defined(BOOST_NO_STDC_NAMESPACE)
namespace std{
using ::size_t;
} // namespace std
#endif

#include <boost/cstdint.hpp>
//#include <boost/limits.hpp>
//#include <boost/io/ios_state.hpp>
#include <boost/scoped_ptr.hpp>
#include <boost/throw_exception.hpp>

#include <boost/archive/basic_streambuf_locale_saver.hpp>
#include <boost/archive/archive_exception.hpp>
#include <boost/archive/detail/auto_link_archive.hpp>
#include <boost/serialization/is_bitwise_serializable.hpp>
#include <boost/mpl/placeholders.hpp>
#include <boost/serialization/array.hpp>
#include <boost/archive/detail/abi_prefix.hpp> // must be the last
header

namespace boost {
namespace archive {

/////////////////////////////////////////////////////////////////////////
// class basic_binary_oprimitive - binary output of prmitives

template<class Archive, class Elem, class Tr>
class basic_binary_oprimitive
{
#ifndef BOOST_NO_MEMBER_TEMPLATE_FRIENDS
friend class save_access;
protected:
#else
public:
#endif
std::basic_streambuf<Elem, Tr> & m_sb;
// return a pointer to the most derived class
Archive * This(){
return static_cast<Archive *>(this);
}
boost::scoped_ptr<std::locale> archive_locale;
basic_streambuf_locale_saver<Elem, Tr> locale_saver;

// default saving of primitives.
template<class T>
void save(const T & t)
{
save_binary(& t, sizeof(T));
}

/////////////////////////////////////////////////////////
// fundamental types that need special treatment

// trap usage of invalid uninitialized boolean which would
// otherwise crash on load.
void save(const bool t){
int i = t;
assert(0 == i || 1 == i);
save_binary(& t, sizeof(t));
}
BOOST_ARCHIVE_OR_WARCHIVE_DECL(void)
save(const std::string &s);
#ifndef BOOST_NO_STD_WSTRING
BOOST_ARCHIVE_OR_WARCHIVE_DECL(void)
save(const std::wstring &ws);
#endif
BOOST_ARCHIVE_OR_WARCHIVE_DECL(void)
save(const char * t);
BOOST_ARCHIVE_OR_WARCHIVE_DECL(void)
save(const wchar_t * t);

BOOST_ARCHIVE_OR_WARCHIVE_DECL(void)
init();

BOOST_ARCHIVE_OR_WARCHIVE_DECL(BOOST_PP_EMPTY())
basic_binary_oprimitive(
std::basic_streambuf<Elem, Tr> & sb,
bool no_codecvt
);
BOOST_ARCHIVE_OR_WARCHIVE_DECL(BOOST_PP_EMPTY())
~basic_binary_oprimitive();
public:

// we provide an optimized save for all fundamental types
// typedef serialization::is_bitwise_serializable<mpl::_1>
// use_array_optimization;
// workaround without using mpl lambdas
struct use_array_optimization {
template <class T>
struct apply : public serialization::is_bitwise_serializable<T>
{};
};


// the optimized save_array dispatches to save_binary
template <class ValueType>
void save_array(serialization::array<ValueType> const& a, unsigned
int)
{
save_binary(a.address(),a.count()*sizeof(ValueType ));
}

void save_binary(const void *address, std::size_t count);
};

template<class Archive, class Elem, class Tr>
inline void
basic_binary_oprimitive<Archive, Elem, Tr>::save_binary(
const void *address,
std::size_t count
){
//assert(
//
static_cast<std::size_t>((std::numeric_limits<std: :streamsize>::max)
()) >= count
//);
// note: if the following assertions fail
// a likely cause is that the output stream is set to "text"
// mode where by cr characters recieve special treatment.
// be sure that the output stream is opened with ios::binary
//if(os.fail())
//
boost::throw_exception(archive_exception(archive_e xception::stream_error));
// figure number of elements to output - round up
count = ( count + sizeof(Elem) - 1)
/ sizeof(Elem);
std::streamsize scount = m_sb.sputn(
static_cast<const Elem *>(address),
count
);
if(count != static_cast<std::size_t>(scount))

boost::throw_exception(archive_exception(archive_e xception::stream_error));
//os.write(
// static_cast<const BOOST_DEDUCED_TYPENAME OStream::char_type
*>(address),
// count
//);
//assert(os.good());
}

} //namespace boost
} //namespace archive

#include <boost/archive/detail/abi_suffix.hpp> // pop pragams

#endif // BOOST_ARCHIVE_BASIC_BINARY_OPRIMITIVE_HPP




I guess it boils down to a call of basic_streambuf<>::sputn(). Is
that any
better IYO than memcpy?

Brian Wood
Ebenezer Enterprises
www.webEbenezer.net
 
Reply With Quote
 
 
 
 
James Kanze
Guest
Posts: n/a
 
      05-27-2008
On May 26, 8:15 pm, (E-Mail Removed) wrote:
> On May 22, 1:58 am, James Kanze <(E-Mail Removed)> wrote:


[...]
> In Boost 1.35 they've added an optimization to take advantage of
> contiguous collections of primitive data types. Here is a copy
> of a file that is involved:


Note however:

[...]
> // archives stored as native binary - this should be the fastest way
> // to archive the state of a group of obects. It makes no attempt to
> // convert to any canonical form.


> // IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
> // ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON


Where "same platform" here means compiled on the same hardware,
using the same version of the same compiler, and the same
compiler options. If you ever recompile your executable with a
more recent version of the compiler, or with different options,
you may no longer be able to read the data.

In sum, it's an acceptable solution for temporary files within a
single run of the executable, but not for much else.

--
James Kanze (GABI Software) email:(E-Mail Removed)
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
 
Reply With Quote
 
 
 
 
gpderetta
Guest
Posts: n/a
 
      05-27-2008
On May 27, 10:04*am, James Kanze <(E-Mail Removed)> wrote:
> On May 26, 8:15 pm, (E-Mail Removed) wrote:
>
> > On May 22, 1:58 am, James Kanze <(E-Mail Removed)> wrote:

>
> * * [...]
>
> > In Boost 1.35 they've added an optimization to take advantage of
> > contiguous collections of primitive data types. *Here is a copy
> > of a file that is involved:

>
> Note however:
>
> * * [...]
>
> > // archives stored as native binary - this should be the fastest way
> > // to archive the state of a group of obects. *It makes no attempt to
> > // convert to any canonical form.
> > // IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
> > // ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON

>
> Where "same platform" here means compiled on the same hardware,
> using the same version of the same compiler, and the same
> compiler options. *If you ever recompile your executable with a
> more recent version of the compiler, or with different options,
> you may no longer be able to read the data.
>
> In sum, it's an acceptable solution for temporary files within a
> single run of the executable, but not for much else.


Modulo what is guaranteed by the compiler/platform ABI, I guess.

In particular, the Boost.Serialization binary format is primarily used
by Boost.MPI (which obviously is a wrapper around MPI) for inter
process communication. I think that the idea is that the MPI layer
will take care of marshaling between peers and thus resolve any
representation difference. I think that in practice most (but not all)
MPI implementations just assume that peers use the same layout format
(i.e. same CPU/compiler/OS) and just network copy bytes back and
forward.

In a sense the distributed program is a logical single run of the same
program even if in practice are different processes running on
different machines, so your observation is still valid

--
Giovanni P. Deretta
 
Reply With Quote
 
James Kanze
Guest
Posts: n/a
 
      05-28-2008
On May 27, 12:07 pm, gpderetta <(E-Mail Removed)> wrote:
> On May 27, 10:04 am, James Kanze <(E-Mail Removed)> wrote:
> > On May 26, 8:15 pm, (E-Mail Removed) wrote:


> > > On May 22, 1:58 am, James Kanze <(E-Mail Removed)> wrote:


> > [...]


> > > In Boost 1.35 they've added an optimization to take advantage of
> > > contiguous collections of primitive data types. Here is a copy
> > > of a file that is involved:


> > Note however:


> > [...]


> > > // archives stored as native binary - this should be the fastest way
> > > // to archive the state of a group of obects. It makes no attempt to
> > > // convert to any canonical form.
> > > // IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
> > > // ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON


> > Where "same platform" here means compiled on the same hardware,
> > using the same version of the same compiler, and the same
> > compiler options. If you ever recompile your executable with a
> > more recent version of the compiler, or with different options,
> > you may no longer be able to read the data.


> > In sum, it's an acceptable solution for temporary files within a
> > single run of the executable, but not for much else.


> Modulo what is guaranteed by the compiler/platform ABI, I guess.


Supposing you can trust them to be stable. In actual
practice, I've seen plenty of size changes, and I've seen long
and the floating point types change their representation, just
between different versions of the compiler. Not to mention
changes in padding which, at least in some cases depend on
compiler options. (For that matter, on most of the machines I
use, the size of a long depends on compiler options. And is the
sort of option that someone is likely to change in the makefile,
because e.g. they suddenly have to deal with big files.)

> In particular, the Boost.Serialization binary format is
> primarily used by Boost.MPI (which obviously is a wrapper
> around MPI) for inter process communication. I think that the
> idea is that the MPI layer will take care of marshaling
> between peers and thus resolve any representation difference.
> I think that in practice most (but not all) MPI
> implementations just assume that peers use the same layout
> format (i.e. same CPU/compiler/OS) and just network copy bytes
> back and forward.


> In a sense the distributed program is a logical single run of
> the same program even if in practice are different processes
> running on different machines, so your observation is still
> valid


If the programs are not running on different machines, what's
the point of marshalling. Just put the objects in shared
memory. Marshalling is only necessary if the data is to be used
in a different place or time (networking or persistency). And a
different place or time means a different machine (sooner or
later, in the case of time).

--
James Kanze (GABI Software) email:(E-Mail Removed)
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
 
Reply With Quote
 
gpderetta
Guest
Posts: n/a
 
      05-28-2008
On May 28, 10:30*am, James Kanze <(E-Mail Removed)> wrote:
> On May 27, 12:07 pm, gpderetta <(E-Mail Removed)> wrote:
>
>
>
> > On May 27, 10:04 am, James Kanze <(E-Mail Removed)> wrote:
> > > On May 26, 8:15 pm, (E-Mail Removed) wrote:
> > > > On May 22, 1:58 am, James Kanze <(E-Mail Removed)> wrote:
> > > * * [...]
> > > > In Boost 1.35 they've added an optimization to take advantage of
> > > > contiguous collections of primitive data types. *Here is a copy
> > > > of a file that is involved:
> > > Note however:
> > > * * [...]
> > > > // archives stored as native binary - this should be the fastest way
> > > > // to archive the state of a group of obects. *It makes no attempt to
> > > > // convert to any canonical form.
> > > > // IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
> > > > // ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON
> > > Where "same platform" here means compiled on the same hardware,
> > > using the same version of the same compiler, and the same
> > > compiler options. *If you ever recompile your executable with a
> > > more recent version of the compiler, or with different options,
> > > you may no longer be able to read the data.
> > > In sum, it's an acceptable solution for temporary files within a
> > > single run of the executable, but not for much else.

> > Modulo what is guaranteed by the compiler/platform ABI, I guess.

>
> Supposing you can trust them to be stable. *In actual
> practice, I've seen plenty of size changes, and I've seen long
> and the floating point types change their representation, just
> between different versions of the compiler. *Not to mention
> changes in padding which, at least in some cases depend on
> compiler options. *(For that matter, on most of the machines I
> use, the size of a long depends on compiler options. *And is the
> sort of option that someone is likely to change in the makefile,
> because e.g. they suddenly have to deal with big files.)


The size of long or that of off_t?

>
> > In particular, the Boost.Serialization binary format is
> > primarily used by Boost.MPI (which obviously is a wrapper
> > around MPI) for inter process communication. I think that the
> > idea is that the MPI layer will take care of marshaling
> > between peers and thus resolve any representation difference.
> > I think that in practice most (but not all) MPI
> > implementations just assume that peers use the same layout
> > format (i.e. same CPU/compiler/OS) and just network copy bytes
> > back and forward.
> > In a sense the distributed program is a logical single run of
> > the same program even if in practice are different processes
> > running on different machines, so your observation is still
> > valid

>
> If the programs are not running on different machines, what's
> the point of marshalling. *Just put the objects in shared
> memory. *Marshalling is only necessary if the data is to be used
> in a different place or time (networking or persistency). *And a
> different place or time means a different machine (sooner or
> later, in the case of time).
>


Well, MPI programs runs on large clusters of, usually, homogeneous
machines, connected via LAN. The same program will spawn
multiple copies of itself on every machine in the cluster, and every
copy communicates via message passing.
So you have one logical program which is partitioned on multiple
machines. I guess that most MPI implementations do not bother (in fact
I do not even know if it is required by the standard) to convert
messages to a machine agnostic format before sending it to another
peer.

--
Giovanni P. Deretta
 
Reply With Quote
 
James Kanze
Guest
Posts: n/a
 
      05-28-2008
On May 28, 12:11 pm, gpderetta <(E-Mail Removed)> wrote:
> On May 28, 10:30 am, James Kanze <(E-Mail Removed)> wrote:
> > On May 27, 12:07 pm, gpderetta <(E-Mail Removed)> wrote:


> > > On May 27, 10:04 am, James Kanze <(E-Mail Removed)> wrote:
> > > > On May 26, 8:15 pm, (E-Mail Removed) wrote:
> > > > > On May 22, 1:58 am, James Kanze <(E-Mail Removed)> wrote:
> > > > [...]
> > > > > In Boost 1.35 they've added an optimization to take advantage of
> > > > > contiguous collections of primitive data types. Here is a copy
> > > > > of a file that is involved:
> > > > Note however:
> > > > [...]
> > > > > // archives stored as native binary - this should be the fastest way
> > > > > // to archive the state of a group of obects. It makes no attempt to
> > > > > // convert to any canonical form.
> > > > > // IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
> > > > > // ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON
> > > > Where "same platform" here means compiled on the same hardware,
> > > > using the same version of the same compiler, and the same
> > > > compiler options. If you ever recompile your executable with a
> > > > more recent version of the compiler, or with different options,
> > > > you may no longer be able to read the data.
> > > > In sum, it's an acceptable solution for temporary files within a
> > > > single run of the executable, but not for much else.
> > > Modulo what is guaranteed by the compiler/platform ABI, I guess.


> > Supposing you can trust them to be stable. In actual
> > practice, I've seen plenty of size changes, and I've seen long
> > and the floating point types change their representation, just
> > between different versions of the compiler. Not to mention
> > changes in padding which, at least in some cases depend on
> > compiler options. (For that matter, on most of the machines I
> > use, the size of a long depends on compiler options. And is the
> > sort of option that someone is likely to change in the makefile,
> > because e.g. they suddenly have to deal with big files.)


> The size of long or that of off_t?


No matter. The point is that they have to compile with
different options, and suddenly long has changed its size.

> > > In particular, the Boost.Serialization binary format is
> > > primarily used by Boost.MPI (which obviously is a wrapper
> > > around MPI) for inter process communication. I think that
> > > the idea is that the MPI layer will take care of
> > > marshaling between peers and thus resolve any
> > > representation difference. I think that in practice most
> > > (but not all) MPI implementations just assume that peers
> > > use the same layout format (i.e. same CPU/compiler/OS) and
> > > just network copy bytes back and forward. In a sense the
> > > distributed program is a logical single run of the same
> > > program even if in practice are different processes
> > > running on different machines, so your observation is
> > > still valid


> > If the programs are not running on different machines,
> > what's the point of marshalling. Just put the objects in
> > shared memory. Marshalling is only necessary if the data is
> > to be used in a different place or time (networking or
> > persistency). And a different place or time means a
> > different machine (sooner or later, in the case of time).


> Well, MPI programs runs on large clusters of, usually,
> homogeneous machines, connected via LAN.


That's original. I don't think I've ever seen a cluster of
machines where every system in the cluster was identical. At
the very least, you'll have different versions of Sparc, or PC.
Some of which are 32 bit, and others 64. The cluster may start
out homogeneous, but one of the machines breaks down, and is
replaced with a newer model...

The real question, however, doesn't concern just the machines.
If all of the machines are running a single executable, loaded
from the same shared disk, it will probably work. If not, then
sooner or later, some of the machines will have different
compiles of the program, which may or may not be binary
compatible. In practice, the old rule always holds: identical
copies aren't. (Remember, binary compatibility can be lost just
by changing options, or using a newer version of the compiler.)

> The same program will spawn multiple copies of itself on every
> machine in the cluster, and every copy communicates via
> message passing. So you have one logical program which is
> partitioned on multiple machines. I guess that most MPI
> implementations do not bother (in fact I do not even know if
> it is required by the standard) to convert messages to a
> machine agnostic format before sending it to another peer.


Well, I don't know much about that context. In my work, we have
a hetrogeneous network, with PC's under Windows as clients, and
either PC's under Linux or Sparcs under Solaris as servers (and
high level clients). And that more or less corresponds to what
I've seen elswhere as well.

--
James Kanze (GABI Software) email:(E-Mail Removed)
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
 
Reply With Quote
 
gpderetta
Guest
Posts: n/a
 
      05-30-2008
On May 28, 10:06 pm, James Kanze <(E-Mail Removed)> wrote:
> On May 28, 12:11 pm, gpderetta <(E-Mail Removed)> wrote:
>
> > On May 28, 10:30 am, James Kanze <(E-Mail Removed)> wrote:
> > > On May 27, 12:07 pm, gpderetta <(E-Mail Removed)> wrote:
> > > > In particular, the Boost.Serialization binary format is
> > > > primarily used by Boost.MPI (which obviously is a wrapper
> > > > around MPI) for inter process communication. I think that
> > > > the idea is that the MPI layer will take care of
> > > > marshaling between peers and thus resolve any
> > > > representation difference. I think that in practice most
> > > > (but not all) MPI implementations just assume that peers
> > > > use the same layout format (i.e. same CPU/compiler/OS) and
> > > > just network copy bytes back and forward. In a sense the
> > > > distributed program is a logical single run of the same
> > > > program even if in practice are different processes
> > > > running on different machines, so your observation is
> > > > still valid
> > > If the programs are not running on different machines,
> > > what's the point of marshalling. Just put the objects in
> > > shared memory. Marshalling is only necessary if the data is
> > > to be used in a different place or time (networking or
> > > persistency). And a different place or time means a
> > > different machine (sooner or later, in the case of time).

> > Well, MPI programs runs on large clusters of, usually,
> > homogeneous machines, connected via LAN.

>
> That's original. I don't think I've ever seen a cluster of
> machines where every system in the cluster was identical.


I think that for MPI it is common. Some vendors even sell shrink
wrapped clusters in a box (something like a big closet with thousands
of different computers-on-a-board, each running a different OS image).
Even custom built MPI clusters are fairly homogeneous (i.e. at least
same architecture and OS version).

I think that you work mostly on services applications, while MPI is
more common on high performance computing.

> [...]
> The real question, however, doesn't concern just the machines.
> If all of the machines are running a single executable, loaded
> from the same shared disk, it will probably work. If not, then
> sooner or later, some of the machines will have different
> compiles of the program, which may or may not be binary
> compatible. In practice, the old rule always holds: identical
> copies aren't. (Remember, binary compatibility can be lost just
> by changing options, or using a newer version of the compiler.)
>


Yep, one need to be careful, but at least with the compiler I use,
options that change the ABI are explicitly documented as such.
Probably a much bigger problem are differences in third party
libraries between machines (i.e. do not expect the layout of objects
you do not control to stay stable).

> > The same program will spawn multiple copies of itself on every
> > machine in the cluster, and every copy communicates via
> > message passing. So you have one logical program which is
> > partitioned on multiple machines. I guess that most MPI
> > implementations do not bother (in fact I do not even know if
> > it is required by the standard) to convert messages to a
> > machine agnostic format before sending it to another peer.

>
> Well, I don't know much about that context. In my work, we have
> a hetrogeneous network, with PC's under Windows as clients, and
> either PC's under Linux or Sparcs under Solaris as servers (and
> high level clients). And that more or less corresponds to what
> I've seen elswhere as well.
>


Where I work, clusters are composed of hundreds of very different
machines, but all use the same architecture and exact same OS version
(so that we can copy binaries around and not have to worry about
library incompatibilities). We do not use MPI though, but have an in-
house communication framework which does take care of marshaling in a
(mostly) system agnostic format.

--
Giovanni P. Deretta



 
Reply With Quote
 
James Kanze
Guest
Posts: n/a
 
      05-31-2008
On May 30, 6:38 pm, gpderetta <(E-Mail Removed)> wrote:
> On May 28, 10:06 pm, James Kanze <(E-Mail Removed)> wrote:
> > On May 28, 12:11 pm, gpderetta <(E-Mail Removed)> wrote:


> > > On May 28, 10:30 am, James Kanze <(E-Mail Removed)> wrote:
> > > > On May 27, 12:07 pm, gpderetta <(E-Mail Removed)> wrote:
> > > > > In particular, the Boost.Serialization binary format is
> > > > > primarily used by Boost.MPI (which obviously is a wrapper
> > > > > around MPI) for inter process communication. I think that
> > > > > the idea is that the MPI layer will take care of
> > > > > marshaling between peers and thus resolve any
> > > > > representation difference. I think that in practice most
> > > > > (but not all) MPI implementations just assume that peers
> > > > > use the same layout format (i.e. same CPU/compiler/OS) and
> > > > > just network copy bytes back and forward. In a sense the
> > > > > distributed program is a logical single run of the same
> > > > > program even if in practice are different processes
> > > > > running on different machines, so your observation is
> > > > > still valid
> > > > If the programs are not running on different machines,
> > > > what's the point of marshalling. Just put the objects in
> > > > shared memory. Marshalling is only necessary if the data is
> > > > to be used in a different place or time (networking or
> > > > persistency). And a different place or time means a
> > > > different machine (sooner or later, in the case of time).
> > > Well, MPI programs runs on large clusters of, usually,
> > > homogeneous machines, connected via LAN.


> > That's original. I don't think I've ever seen a cluster of
> > machines where every system in the cluster was identical.


> I think that for MPI it is common. Some vendors even sell
> shrink wrapped clusters in a box (something like a big closet
> with thousands of different computers-on-a-board, each running
> a different OS image). Even custom built MPI clusters are
> fairly homogeneous (i.e. at least same architecture and OS
> version).


> I think that you work mostly on services applications, while
> MPI is more common on high performance computing.


I realized that much, but I wasn't aware that it was that common
even on high performance computing. The high performance
computing solutions I've seen have mostly involved a lot of
CPU's using the same memory, so marshalling wasn't an issue.
(But I'm not much of an expert in the domain, and I've not seen
that many systems, so what I've seen doesn't mean much.)

> > [...]
> > The real question, however, doesn't concern just the machines.
> > If all of the machines are running a single executable, loaded
> > from the same shared disk, it will probably work. If not, then
> > sooner or later, some of the machines will have different
> > compiles of the program, which may or may not be binary
> > compatible. In practice, the old rule always holds: identical
> > copies aren't. (Remember, binary compatibility can be lost just
> > by changing options, or using a newer version of the compiler.)


> Yep, one need to be careful, but at least with the compiler I use,
> options that change the ABI are explicitly documented as such.


Lucky guy. For the most part, what the options actually do
is well documented, and if you understand a bit about what it
means at the hardware level, you can figure out which ones are
safe, and which aren't. But it's far from explicit.

Note that this can be a problem just trying to statically link
libraries; you don't need marshalling at all to get into
trouble. (Or rather: you don't want to have to marshall every
time you pass an std::vector to a function in another module.)

> Probably a much bigger problem are differences in third party
> libraries between machines (i.e. do not expect the layout of
> objects you do not control to stay stable).


That's another problem entirely, and affects linking more than
marshalling. The problem is that compilers may change
representation between versions, etc.

> > > The same program will spawn multiple copies of itself on every
> > > machine in the cluster, and every copy communicates via
> > > message passing. So you have one logical program which is
> > > partitioned on multiple machines. I guess that most MPI
> > > implementations do not bother (in fact I do not even know if
> > > it is required by the standard) to convert messages to a
> > > machine agnostic format before sending it to another peer.


> > Well, I don't know much about that context. In my work, we have
> > a hetrogeneous network, with PC's under Windows as clients, and
> > either PC's under Linux or Sparcs under Solaris as servers (and
> > high level clients). And that more or less corresponds to what
> > I've seen elswhere as well.


> Where I work, clusters are composed of hundreds of very
> different machines, but all use the same architecture and
> exact same OS version (so that we can copy binaries around and
> not have to worry about library incompatibilities). We do not
> use MPI though, but have an in- house communication framework
> which does take care of marshaling in a (mostly) system
> agnostic format.


Yes. We do something more or less like this for the clients:
they're all PC's under Windows, and we use a lowest common
denominator which should work for all Windows systems. Our
machines are geographically distributed, however, so
realistically, ensuring exactly the same version of the OS,
isn't possible.

For the servers, economic considerations result in a decision to
move from Solaris on Sparc to Linux on PC, at least for all but
the most critical systems. Similarly, economic considerations
mean that the entire park won't be upgraded at the same instant.
Are you saying that if a decision comes to upgrade the
architecture, you change all of the machines in a cluster at
once? (But maybe... I can imagine that all of the machines in a
cluster still cost less than one supercomputer. And if you were
using a supercomputer, and wanted to upgrade, you'd change it
all at once. I guess it's just a different mindset.)

--
James Kanze (GABI Software) email:(E-Mail Removed)
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
reading binary file into memory. Converting from char to uint32,float, double, ASCII strings etc (static_cast< > ?) someone C++ 37 10-18-2011 01:32 AM
float to string to float, with first float == second float Carsten Fuchs C++ 45 10-08-2009 09:47 AM
(const char *cp) and (char *p) are consistent type, (const char **cpp) and (char **pp) are not consistent lovecreatesbeauty C Programming 1 05-09-2006 08:01 AM
/usr/bin/ld: ../../dist/lib/libjsdombase_s.a(BlockGrouper.o)(.text+0x98): unresolvable relocation against symbol `std::basic_ostream<char, std::char_traits<char> >& std::endl<char, std::char_traits<char> >(std::basic_ostre silverburgh.meryl@gmail.com C++ 3 03-09-2006 12:14 AM
Re: float->byte->float is same with original float image. why float->ubyte->float is different??? bd C Programming 0 07-07-2003 12:09 AM



Advertisments