Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   C++ (http://www.velocityreviews.com/forums/f39-c.html)
-   -   reinterpret_cast portability/alignment issues (http://www.velocityreviews.com/forums/t459552-reinterpret_cast-portability-alignment-issues.html)

Lionel B 12-31-2006 01:16 PM

reinterpret_cast portability/alignment issues
 
Greetings,

I have some code that is to read unformatted data from disc and interpret
it as blocks of unsigned integers. In an attempt to achieve efficiency
(it is pretty essential for my application that the code be
speed optimized[*]) I use reinterpret_cast to alias a block of chars read
in from disc as a block of integer "words". My existing code (see
simplified code below) appears to work well enough on the platforms
available to me, but I would like to achieve maximum portability and would
really appreciate any commentary on possible portability (and also memory
alignment) issues that might arise from my approach.

I understand that in some sense all bets are off when using
reinterpret_cast as details will be implementation-defined but I can't see
why my aliasing method (with suitable checks) should cause any problems.

As to memory alignment for efficient access to my memory buffer, I'm not
sure how global operator new handles memory alignment (I shouldn't think
the standard has anything to say about this), so I might be prepared to
either (i) "manually" align my memory blocks - not actually sure how to do
this or how much portability might be achieved with this approach) - or
(ii) use malloc / memalign, whatever, from the (or rather "a") C library,
where alignment behaviour might be more precisely specified (eg the GNU C
library). Of course there will be portability issues with this approach.

In essence my problem seems to involve a potential trade-off between
efficiency and portability. Again, any comments, suggestions, alternative
approaches welcome.

Simplified code below.
[*] The code to be used in anger is to grab random numbers from huge
binary data files for a statistical application ... I have identified
random number manipulation as a time bottleneck.

--- BEGIN CODE ----

// main.cpp

#include <iostream>
#include <fstream>
#include <string>

// Read unformatted data from disc and interpret as unsigned integers

int main()
{
// the word type (an unsigned integer type - season to taste)
typedef unsigned long word_t;

// number of chars per word
const size_t wordsize = sizeof(word_t);
std::cout << "wordsize = " << wordsize << '\n';

// data file (we open with ios::ate since we want to get file size first)
const std::string bitsfile = "bits.dat";
std::ifstream bfs(bitsfile.c_str(),std::ios::in|std::ios::binary |std::ios::ate);
if (!bfs.is_open()) {
std::cerr << "failed to open file " << bitsfile << '\n';
return 1;
}

// get file size
const size_t cfilesize = bfs.tellg();
std::cout << "filesize = " << cfilesize << " chars\n";

// number of bytes (chars) to read - mustn't be bigger than file size!
const size_t cblocksize = 128;
std::cout << "block size (char) = " << cblocksize << '\n';
if (cblocksize>cfilesize) {
std::cerr << "block size cannnot be larger than file size\n";
return 1;
}

// number of words to write - must be divisible by word size!
const size_t wblocksize = cblocksize/wordsize;
std::cout << "block size (word) = " << wblocksize << '\n';
if (wblocksize*wordsize != cblocksize) {
std::cerr << "block size must be divisible by word size\n";
return 1;
}

// char buffer for reading
// COMMENTS? alignement issues?
char* const cbuf(new char[cblocksize]);

// word buffer for writing aliased to char buffer
// COMMENTS? how portable is this...?
// COMMENTS? alignement issues?
const word_t* const wbuf(reinterpret_cast<word_t* const>(cbuf));

// seek to beginning of file and read block of chars
bfs.seekg(0);
bfs.read(cbuf,cblocksize);
bfs.close();

// write out words
for (size_t i;i<wblocksize;++i) std::cout << "word " << i << " = " << wbuf[i] << '\n';

// clean up
delete [] cbuf;

return 0;
}

--- END CODE ----

--
Lionel B

Moonlit 12-31-2006 02:43 PM

Re: reinterpret_cast portability/alignment issues
 
Hi

I didn't go through all your code but some quick nodes
I would new it as a 'word ' array not as a 'character' array just to make
sure it is at the right boundary. You can the reinterpret_cast to char* and
now that it is correctly alligned.

There are usually compiler options you can set. In visual studio you can set
aliignement for your project or add pragma ... directives in different
places.

If you write and read from a systems with different architectures
(big/little endian) your code of course will fail

BTW does the getfilesize really work (shouldn't you relocate the get ptr to
the end of file first)?

Regards, Ron AF Greve

http://moonlit.xs4all.nl

"Lionel B" <me@privacy.net> wrote in message
news:KKOlh.43592$UC.7889@newsfe5-win.ntli.net...
> Greetings,
>
> I have some code that is to read unformatted data from disc and interpret
> it as blocks of unsigned integers. In an attempt to achieve efficiency
> (it is pretty essential for my application that the code be
> speed optimized[*]) I use reinterpret_cast to alias a block of chars read
> in from disc as a block of integer "words". My existing code (see
> simplified code below) appears to work well enough on the platforms
> available to me, but I would like to achieve maximum portability and would
> really appreciate any commentary on possible portability (and also memory
> alignment) issues that might arise from my approach.
>
> I understand that in some sense all bets are off when using
> reinterpret_cast as details will be implementation-defined but I can't see
> why my aliasing method (with suitable checks) should cause any problems.
>
> As to memory alignment for efficient access to my memory buffer, I'm not
> sure how global operator new handles memory alignment (I shouldn't think
> the standard has anything to say about this), so I might be prepared to
> either (i) "manually" align my memory blocks - not actually sure how to do
> this or how much portability might be achieved with this approach) - or
> (ii) use malloc / memalign, whatever, from the (or rather "a") C library,
> where alignment behaviour might be more precisely specified (eg the GNU C
> library). Of course there will be portability issues with this approach.
>
> In essence my problem seems to involve a potential trade-off between
> efficiency and portability. Again, any comments, suggestions, alternative
> approaches welcome.
>
> Simplified code below.
>
>[*] The code to be used in anger is to grab random numbers from huge
> binary data files for a statistical application ... I have identified
> random number manipulation as a time bottleneck.
>
> --- BEGIN CODE ----
>
> // main.cpp
>
> #include <iostream>
> #include <fstream>
> #include <string>
>
> // Read unformatted data from disc and interpret as unsigned integers
>
> int main()
> {
> // the word type (an unsigned integer type - season to taste)
> typedef unsigned long word_t;
>
> // number of chars per word
> const size_t wordsize = sizeof(word_t);
> std::cout << "wordsize = " << wordsize << '\n';
>
> // data file (we open with ios::ate since we want to get file size first)
> const std::string bitsfile = "bits.dat";
> std::ifstream
> bfs(bitsfile.c_str(),std::ios::in|std::ios::binary |std::ios::ate);
> if (!bfs.is_open()) {
> std::cerr << "failed to open file " << bitsfile << '\n';
> return 1;
> }
>
> // get file size
> const size_t cfilesize = bfs.tellg();
> std::cout << "filesize = " << cfilesize << " chars\n";
>
> // number of bytes (chars) to read - mustn't be bigger than file size!
> const size_t cblocksize = 128;
> std::cout << "block size (char) = " << cblocksize << '\n';
> if (cblocksize>cfilesize) {
> std::cerr << "block size cannnot be larger than file size\n";
> return 1;
> }
>
> // number of words to write - must be divisible by word size!
> const size_t wblocksize = cblocksize/wordsize;
> std::cout << "block size (word) = " << wblocksize << '\n';
> if (wblocksize*wordsize != cblocksize) {
> std::cerr << "block size must be divisible by word size\n";
> return 1;
> }
>
> // char buffer for reading
> // COMMENTS? alignement issues?
> char* const cbuf(new char[cblocksize]);
>
> // word buffer for writing aliased to char buffer
> // COMMENTS? how portable is this...?
> // COMMENTS? alignement issues?
> const word_t* const wbuf(reinterpret_cast<word_t* const>(cbuf));
>
> // seek to beginning of file and read block of chars
> bfs.seekg(0);
> bfs.read(cbuf,cblocksize);
> bfs.close();
>
> // write out words
> for (size_t i;i<wblocksize;++i) std::cout << "word " << i << " = " <<
> wbuf[i] << '\n';
>
> // clean up
> delete [] cbuf;
>
> return 0;
> }
>
> --- END CODE ----
>
> --
> Lionel B




Ron Natalie 12-31-2006 02:45 PM

Re: reinterpret_cast portability/alignment issues
 
Lionel B wrote:

>
> // char buffer for reading
> // COMMENTS? alignement issues?
> char* const cbuf(new char[cblocksize]);
>
> // word buffer for writing aliased to char buffer
> // COMMENTS? how portable is this...?
> // COMMENTS? alignement issues?
> const word_t* const wbuf(reinterpret_cast<word_t* const>(cbuf));
>

If you're casting the direct return from new (as you are here), it
should be OK. new char[] is required to return a "universally"
aligned thing as long as you allocate at least as much as the
thing you are aligning to.

Of course, the universal guarantee is you can always alias things
to chars and they work. So rather than allocating a buffer of
char, you can allocate a buffer of something else and then
cast it to char to read/copy-in the data.

Moonlit 12-31-2006 02:45 PM

Re: reinterpret_cast portability/alignment issues
 
Oops don't mind the BTW missed the ate and the comment :-(



Regards, Ron AF Greve

http://moonlit.xs4all.nl

"Moonlit" <news moonlit xs4all nl> wrote in message
news:4597ccac$0$334$e4fe514c@news.xs4all.nl...
> Hi
>
> I didn't go through all your code but some quick nodes
> I would new it as a 'word ' array not as a 'character' array just to make
> sure it is at the right boundary. You can the reinterpret_cast to char*
> and now that it is correctly alligned.
>
> There are usually compiler options you can set. In visual studio you can
> set aliignement for your project or add pragma ... directives in different
> places.
>
> If you write and read from a systems with different architectures
> (big/little endian) your code of course will fail
>
> BTW does the getfilesize really work (shouldn't you relocate the get ptr
> to the end of file first)?
>
> Regards, Ron AF Greve
>
> http://moonlit.xs4all.nl
>
> "Lionel B" <me@privacy.net> wrote in message
> news:KKOlh.43592$UC.7889@newsfe5-win.ntli.net...
>> Greetings,
>>
>> I have some code that is to read unformatted data from disc and interpret
>> it as blocks of unsigned integers. In an attempt to achieve efficiency
>> (it is pretty essential for my application that the code be
>> speed optimized[*]) I use reinterpret_cast to alias a block of chars
>> read
>> in from disc as a block of integer "words". My existing code (see
>> simplified code below) appears to work well enough on the platforms
>> available to me, but I would like to achieve maximum portability and
>> would
>> really appreciate any commentary on possible portability (and also memory
>> alignment) issues that might arise from my approach.
>>
>> I understand that in some sense all bets are off when using
>> reinterpret_cast as details will be implementation-defined but I can't
>> see
>> why my aliasing method (with suitable checks) should cause any problems.
>>
>> As to memory alignment for efficient access to my memory buffer, I'm not
>> sure how global operator new handles memory alignment (I shouldn't think
>> the standard has anything to say about this), so I might be prepared to
>> either (i) "manually" align my memory blocks - not actually sure how to
>> do
>> this or how much portability might be achieved with this approach) - or
>> (ii) use malloc / memalign, whatever, from the (or rather "a") C library,
>> where alignment behaviour might be more precisely specified (eg the GNU C
>> library). Of course there will be portability issues with this approach.
>>
>> In essence my problem seems to involve a potential trade-off between
>> efficiency and portability. Again, any comments, suggestions, alternative
>> approaches welcome.
>>
>> Simplified code below.
>>
>>[*] The code to be used in anger is to grab random numbers from huge
>> binary data files for a statistical application ... I have identified
>> random number manipulation as a time bottleneck.
>>
>> --- BEGIN CODE ----
>>
>> // main.cpp
>>
>> #include <iostream>
>> #include <fstream>
>> #include <string>
>>
>> // Read unformatted data from disc and interpret as unsigned integers
>>
>> int main()
>> {
>> // the word type (an unsigned integer type - season to taste)
>> typedef unsigned long word_t;
>>
>> // number of chars per word
>> const size_t wordsize = sizeof(word_t);
>> std::cout << "wordsize = " << wordsize << '\n';
>>
>> // data file (we open with ios::ate since we want to get file size first)
>> const std::string bitsfile = "bits.dat";
>> std::ifstream
>> bfs(bitsfile.c_str(),std::ios::in|std::ios::binary |std::ios::ate);
>> if (!bfs.is_open()) {
>> std::cerr << "failed to open file " << bitsfile << '\n';
>> return 1;
>> }
>>
>> // get file size
>> const size_t cfilesize = bfs.tellg();
>> std::cout << "filesize = " << cfilesize << " chars\n";
>>
>> // number of bytes (chars) to read - mustn't be bigger than file size!
>> const size_t cblocksize = 128;
>> std::cout << "block size (char) = " << cblocksize << '\n';
>> if (cblocksize>cfilesize) {
>> std::cerr << "block size cannnot be larger than file size\n";
>> return 1;
>> }
>>
>> // number of words to write - must be divisible by word size!
>> const size_t wblocksize = cblocksize/wordsize;
>> std::cout << "block size (word) = " << wblocksize << '\n';
>> if (wblocksize*wordsize != cblocksize) {
>> std::cerr << "block size must be divisible by word size\n";
>> return 1;
>> }
>>
>> // char buffer for reading
>> // COMMENTS? alignement issues?
>> char* const cbuf(new char[cblocksize]);
>>
>> // word buffer for writing aliased to char buffer
>> // COMMENTS? how portable is this...?
>> // COMMENTS? alignement issues?
>> const word_t* const wbuf(reinterpret_cast<word_t* const>(cbuf));
>>
>> // seek to beginning of file and read block of chars
>> bfs.seekg(0);
>> bfs.read(cbuf,cblocksize);
>> bfs.close();
>>
>> // write out words
>> for (size_t i;i<wblocksize;++i) std::cout << "word " << i << " = " <<
>> wbuf[i] << '\n';
>>
>> // clean up
>> delete [] cbuf;
>>
>> return 0;
>> }
>>
>> --- END CODE ----
>>
>> --
>> Lionel B

>
>




Lionel B 01-01-2007 03:11 PM

Re: reinterpret_cast portability/alignment issues
 
On Sun, 31 Dec 2006 15:43:55 +0100, Moonlit wrote:

> Hi
>
> I didn't go through all your code but some quick nodes
> I would new it as a 'word ' array not as a 'character' array just to make
> sure it is at the right boundary. You can the reinterpret_cast to char* and
> now that it is correctly alligned.


Right. So if I do:

word_t* const wbuf(new word_t[wblocksize]);
const char* const cbuf(reinterpret_cast<char* const>(wbuf));

then wbuf will be guaranteed to be aligned on a word boundary? If so, then
presumably the following:

> There are usually compiler options you can set. In visual studio you can set
> aliignement for your project or add pragma ... directives in different
> places.


would not then be relevant.

> If you write and read from a systems with different architectures
> (big/little endian) your code of course will fail


I am aware of that - I will probably find/write a utility to reverse the
endianess of the raw data on disc as the simplest approach acceptable for
my app.

Thanks,

--
Lionel B

Lionel B 01-01-2007 03:12 PM

Re: reinterpret_cast portability/alignment issues
 
On Sun, 31 Dec 2006 09:45:18 -0500, Ron Natalie wrote:

> Lionel B wrote:
>
>>
>> // char buffer for reading
>> // COMMENTS? alignement issues?
>> char* const cbuf(new char[cblocksize]);
>>
>> // word buffer for writing aliased to char buffer
>> // COMMENTS? how portable is this...?
>> // COMMENTS? alignement issues?
>> const word_t* const wbuf(reinterpret_cast<word_t* const>(cbuf));
>>

> If you're casting the direct return from new (as you are here), it
> should be OK. new char[] is required to return a "universally"
> aligned thing as long as you allocate at least as much as the
> thing you are aligning to.


Not sure I understand that...

> Of course, the universal guarantee is you can always alias things
> to chars and they work. So rather than allocating a buffer of
> char, you can allocate a buffer of something else and then
> cast it to char to read/copy-in the data.


Yes, my original code allocated a word_t buffer then aliased it to char*
.... can't remember why I changed it. So if I do it this way:

word_t* const wbuf(new word_t[wblocksize]);
const char* const cbuf(reinterpret_cast<char* const>(wbuf));

does this then guarantee that wbuf will be aligned on a word boundary?

--
Lionel B

Noah Roberts 01-01-2007 06:28 PM

Re: reinterpret_cast portability/alignment issues
 

Lionel B wrote:
> Greetings,
>
> I have some code that is to read unformatted data from disc and interpret
> it as blocks of unsigned integers. In an attempt to achieve efficiency
> (it is pretty essential for my application that the code be
> speed optimized[*]) I use reinterpret_cast to alias a block of chars read
> in from disc as a block of integer "words". My existing code (see
> simplified code below) appears to work well enough on the platforms
> available to me, but I would like to achieve maximum portability and would
> really appreciate any commentary on possible portability (and also memory
> alignment) issues that might arise from my approach.


The numbers you get back will be completely different if
reading/writing on architectures with different endianness.


Moonlit 01-01-2007 07:43 PM

Re: reinterpret_cast portability/alignment issues
 
Hi,





"Lionel B" <me@privacy.net> wrote in message
news:Pw9mh.4001$696.331@newsfe7-win.ntli.net...
> On Sun, 31 Dec 2006 15:43:55 +0100, Moonlit wrote:
>
>> Hi
>>
>> I didn't go through all your code but some quick nodes
>> I would new it as a 'word ' array not as a 'character' array just to
>> make
>> sure it is at the right boundary. You can the reinterpret_cast to char*
>> and
>> now that it is correctly alligned.

>
> Right. So if I do:
>
> word_t* const wbuf(new word_t[wblocksize]);
> const char* const cbuf(reinterpret_cast<char* const>(wbuf));
>
> then wbuf will be guaranteed to be aligned on a word boundary? If so, then
> presumably the following:
>

Yes
>> There are usually compiler options you can set. In visual studio you can
>> set
>> aliignement for your project or add pragma ... directives in different
>> places.

>


> would not then be relevant.

You sometimes still need to allign an array of characters. For instance say
you have a union of 2 classes (not pointers to) at sometime you want to make
the
reserved room into a real object. Before that as far as I know (but I may be
wrong) you can only allocate them as character arrays.

But there are probably more occasions when you want allignment maybe if you
generated assembly from within you app or something like that.
>
>> If you write and read from a systems with different architectures
>> (big/little endian) your code of course will fail

>
> I am aware of that - I will probably find/write a utility to reverse the
> endianess of the raw data on disc as the simplest approach acceptable for
> my app.
>
> Thanks,
>
> --
> Lionel B


Regards, Ron AF Greve

http://moonlit.xs4all.nl



zengkun100@gmail.com 01-02-2007 06:57 AM

Re: reinterpret_cast portability/alignment issues
 
On Windows platform,you should call Windows API function to make sure
memory correctly aligned,such as VirtualAlloc.
And I think C++'s new operator don't do any alignment job.
"Lionel B д
"
> Greetings,
>
> I have some code that is to read unformatted data from disc and interpret
> it as blocks of unsigned integers. In an attempt to achieve efficiency
> (it is pretty essential for my application that the code be
> speed optimized[*]) I use reinterpret_cast to alias a block of chars read
> in from disc as a block of integer "words". My existing code (see
> simplified code below) appears to work well enough on the platforms
> available to me, but I would like to achieve maximum portability and would
> really appreciate any commentary on possible portability (and also memory
> alignment) issues that might arise from my approach.
>
> I understand that in some sense all bets are off when using
> reinterpret_cast as details will be implementation-defined but I can't see
> why my aliasing method (with suitable checks) should cause any problems.
>
> As to memory alignment for efficient access to my memory buffer, I'm not
> sure how global operator new handles memory alignment (I shouldn't think
> the standard has anything to say about this), so I might be prepared to
> either (i) "manually" align my memory blocks - not actually sure how to do
> this or how much portability might be achieved with this approach) - or
> (ii) use malloc / memalign, whatever, from the (or rather "a") C library,
> where alignment behaviour might be more precisely specified (eg the GNU C
> library). Of course there will be portability issues with this approach.
>
> In essence my problem seems to involve a potential trade-off between
> efficiency and portability. Again, any comments, suggestions, alternative
> approaches welcome.
>
> Simplified code below.
>
>[*] The code to be used in anger is to grab random numbers from huge
> binary data files for a statistical application ... I have identified
> random number manipulation as a time bottleneck.
>
> --- BEGIN CODE ----
>
> // main.cpp
>
> #include <iostream>
> #include <fstream>
> #include <string>
>
> // Read unformatted data from disc and interpret as unsigned integers
>
> int main()
> {
> // the word type (an unsigned integer type - season to taste)
> typedef unsigned long word_t;
>
> // number of chars per word
> const size_t wordsize = sizeof(word_t);
> std::cout << "wordsize = " << wordsize << '\n';
>
> // data file (we open with ios::ate since we want to get file size first)
> const std::string bitsfile = "bits.dat";
> std::ifstream bfs(bitsfile.c_str(),std::ios::in|std::ios::binary |std::ios::ate);
> if (!bfs.is_open()) {
> std::cerr << "failed to open file " << bitsfile << '\n';
> return 1;
> }
>
> // get file size
> const size_t cfilesize = bfs.tellg();
> std::cout << "filesize = " << cfilesize << " chars\n";
>
> // number of bytes (chars) to read - mustn't be bigger than file size!
> const size_t cblocksize = 128;
> std::cout << "block size (char) = " << cblocksize << '\n';
> if (cblocksize>cfilesize) {
> std::cerr << "block size cannnot be larger than file size\n";
> return 1;
> }
>
> // number of words to write - must be divisible by word size!
> const size_t wblocksize = cblocksize/wordsize;
> std::cout << "block size (word) = " << wblocksize << '\n';
> if (wblocksize*wordsize != cblocksize) {
> std::cerr << "block size must be divisible by word size\n";
> return 1;
> }
>
> // char buffer for reading
> // COMMENTS? alignement issues?
> char* const cbuf(new char[cblocksize]);
>
> // word buffer for writing aliased to char buffer
> // COMMENTS? how portable is this...?
> // COMMENTS? alignement issues?
> const word_t* const wbuf(reinterpret_cast<word_t* const>(cbuf));
>
> // seek to beginning of file and read block of chars
> bfs.seekg(0);
> bfs.read(cbuf,cblocksize);
> bfs.close();
>
> // write out words
> for (size_t i;i<wblocksize;++i) std::cout << "word " << i << " = " << wbuf[i] << '\n';
>
> // clean up
> delete [] cbuf;
>
> return 0;
> }
>
> --- END CODE ----
>
> --
> Lionel B



Ron Natalie 01-02-2007 02:00 PM

Re: reinterpret_cast portability/alignment issues
 
Lionel B wrote:
> On Sun, 31 Dec 2006 09:45:18 -0500, Ron Natalie wrote:
>
>> Lionel B wrote:
>>
>>> // char buffer for reading
>>> // COMMENTS? alignement issues?
>>> char* const cbuf(new char[cblocksize]);
>>>
>>> // word buffer for writing aliased to char buffer
>>> // COMMENTS? how portable is this...?
>>> // COMMENTS? alignement issues?
>>> const word_t* const wbuf(reinterpret_cast<word_t* const>(cbuf));
>>>

>> If you're casting the direct return from new (as you are here), it
>> should be OK. new char[] is required to return a "universally"
>> aligned thing as long as you allocate at least as much as the
>> thing you are aligning to.

>
> Not sure I understand that...


If you call new char[n* sizeof (word_t) ]
it will return something guaranteed to meet the word_t requirements.
It's a hokiness added specifically for new char to let you use it
to allocate memory, lets say to further pass to placement new or
do cames like you want.

> does this then guarantee that wbuf will be aligned on a word boundary?
>

Of course. wbuf will be aligned and so with cbuf


All times are GMT. The time now is 01:05 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.