«

Arnold wrote:
> I need to read a binary file and store it into a buffer in memory (system
> has large amount of RAM, 2GB+) then pass it to a function. The function
> accepts input as 32 bit unsigned longs (DWORD). I can pass a max of 512
> words to it at a time. So I would pass them in chunks of 512 words until the
> whole file has been processed. I haven't worked with binary files before so
> I'm confused with how to store the binary file into memory. What sort of
> array do I use? Does C allow char only? Can I declare a DWORD buffer since
> that's what the function is taking as input? Or do I need to know the format
> of the original data that binary file is encoding and store it in that?
> That's the part that is really confusing me.
>
> I believe I'll need to used fread to copy the file to that array. I plan on
> getting the size of file, then determining how many DWORD are present in it
> (for example 9000) and use that my number of object parameter in fread. So
> in this case:
>
> fread(buffer, 4,9000,fp); //each DWORD is 4 bytes, 900 DWORDs in my binary
> file
>
> Is that right?
>
> Once I get the file into the buffer, I can then do a loop where I pass 512
> elements of the array to a function until all 9000 elements are processed. I
> hope that's right. Any other tips on improving speed and efficiency would be
> appreciated. Thanks.

The fastest way to access large binary disk files is to use mapped
files. This is off-topic in this NG. Posix systems use mmap() and
win32 systems use CreateFile/CreateFileMapping/MapViewOfFile.

I've written countless layers for these, it's pretty easy to code an
interface that works the same way for Posix and Windows.

Once that is done, you simply have a pointer and a length and you can
cast the pointer to whatever the data really is and viola.

There are numerous cross platform gotchas but if you create the file on
the same machine you read the file you're safe - but that's true for any
binary file anyway. These are - endianness, alignment, pointer translation.

Memory mapping does have the disadvantage that the size of the mapped
segment of the file is limited to your address space. On 64 bit systems
this limit is unlikely to be a problem, however it seems you're
encroaching on that limit here.