Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > I/O ofa large number of files

Reply
Thread Tools

I/O ofa large number of files

 
 
David
Guest
Posts: n/a
 
      08-01-2005
Hi all:

I am processing a 3D bitmaps(essentially ~1024 2D bitmaps with a size of 1MB
each).

If I want read large amount of radom data from this series, how could I
buffer the file to get optimized performance? With WinXP pro/512MB memories
and no other big programmes running at the same time.

Cheers

David


 
Reply With Quote
 
 
 
 
Eric Sosman
Guest
Posts: n/a
 
      08-01-2005


David wrote:
> Hi all:
>
> I am processing a 3D bitmaps(essentially ~1024 2D bitmaps with a size of 1MB
> each).
>
> If I want read large amount of radom data from this series, how could I
> buffer the file to get optimized performance?


If the access pattern is "radom" there isn't much you
can do. Load as much as you can manage into memory and hope
your next "radom" access is for something already loaded; if
it isn't, throw something away and load the piece you're
trying to get at. The selection of what to pre-load, what
to throw away, and how much to load for an out-of-memory
experience depends on the access patterns. Some patterns
will lend themselves to exploitation, others won't.

The C pieces you'll need will probably be fopen (with
"rb" mode, most likely), fseek, fread, and fclose, along
with the usual memory-management stuff. I mention fclose
because your system may not permit you to have 1024 file
streams open simultaneously; you may need to "multiplex"
the large number of files across a smaller number of FILE*
streams by closing and re-opening as necessary. The
FOPEN_MAX macro in <stdio.h> gives an approximation to the
number of streams you can open simultaneously, but the
value should be treated only as an approximation.

> With WinXP pro/512MB memories
> and no other big programmes running at the same time.


Irrelevant detail. Well, highly relevant in some ways,
but not to your question (if there is one) about the C
programming language. Windows-oriented newsgroups may have
suggestions that go beyond what C itself can provide.

--
http://www.velocityreviews.com/forums/(E-Mail Removed)

 
Reply With Quote
 
 
 
 
jacob navia
Guest
Posts: n/a
 
      08-01-2005
David wrote:
> Hi all:
>
> I am processing a 3D bitmaps(essentially ~1024 2D bitmaps with a size of 1MB
> each).
>
> If I want read large amount of radom data from this series, how could I
> buffer the file to get optimized performance? With WinXP pro/512MB memories
> and no other big programmes running at the same time.
>
> Cheers
>
> David
>
>

Map the whole file into memory (backed up by the OS).
Read the docs for MapViewOfFile API. This will not use
all the RAM of course, but the system will do the paging
for you, what is far more efficient than what you can do
yourself.

This is not a standard C function, and in another operating
systems you may have to use a different approach.

Using standard C functions you can do the same (as Eric
Sossman replied), but it will be less efficient and is not
trivial to develop.

A simpler approach using just fopen would be to allocate 1MB of virtual
memory with malloc(), then read the whole file into it. The system will
do the paging for you in that case. Frequently used pages will be kept
in memory, less frequently used will be eventually be paged out.

jacob
 
Reply With Quote
 
Anonymous 7843
Guest
Posts: n/a
 
      08-01-2005
In article <dclchh$1ku$(E-Mail Removed)>,
David <(E-Mail Removed)> wrote:
>
> I am processing a 3D bitmaps(essentially ~1024 2D bitmaps with a size of 1MB
> each).
>
> If I want read large amount of radom data from this series, how could I
> buffer the file to get optimized performance? With WinXP pro/512MB memories
> and no other big programmes running at the same time.


I'll take a wild guess and say that whatever process is generating
these 3D bitmaps is pretty high-tech and not cheap...perhaps there is
room in the budget for another gig of RAM? Then you could load the
whole 3D image into RAM and still have the original 512 for the OS and
application. Next step would be to upgrade to a processor with large
amounts of cache.

Sure, it would be cool to optimize some graphics-intensive C code to
minimize thrashing but I hope you are being paid well enough that
1G RAM is cheaper than a couple days of your valuable time.
--
7842++
 
Reply With Quote
 
David
Guest
Posts: n/a
 
      08-01-2005
Thank you all for thsoe informatvie and inspiring replies!

Maybe for my purpose, I may opt to buffer a number of files and use a
counter to record the times of individual files being accessed. Once an
unbuffered file is required, it will replace the least accessed file.
I would prefer this easy and platform independent method... coz I know
little APIs...
...

Hope the random statistics would do the jod themselves.

"David" <(E-Mail Removed)> wrote in message
news:dclchh$1ku$(E-Mail Removed)...
> Hi all:
>
> I am processing a 3D bitmaps(essentially ~1024 2D bitmaps with a size of
> 1MB each).
>
> If I want read large amount of radom data from this series, how could I
> buffer the file to get optimized performance? With WinXP pro/512MB
> memories and no other big programmes running at the same time.
>
> Cheers
>
> David
>
>



 
Reply With Quote
 
Jasen Betts
Guest
Posts: n/a
 
      08-04-2005
In article <dclchh$1ku$(E-Mail Removed)>, David wrote:
> Hi all:
>
> I am processing a 3D bitmaps(essentially ~1024 2D bitmaps with a size of 1MB
> each).
>
> If I want read large amount of radom data from this series, how could I
> buffer the file to get optimized performance? With WinXP pro/512MB memories
> and no other big programmes running at the same time.


the answer is process specific - it depends what you're doing with the bitmap.

the easiest solution is probably to put more ram in your PC - if the OS can
handle it.

otherwise you may need to break the bitmap up into chunks which can be
processed independantly.


Bye.
Jasen
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Panasonic trying to disrupt sales the OM-D by offering up rumours ofa new, high-end Panasonic? RichA Digital Photography 3 02-15-2012 01:16 PM
problem with a function template returning a private nested class ofa class template ymost@hotmail.com C++ 2 12-28-2008 10:43 AM
OT: Number Nine, Number Nine, Number Nine FrisbeeŽ MCSE 37 09-26-2005 04:06 PM
dbase: how to get fieldNAME ofa table bart plessers ASP General 4 10-02-2003 11:10 AM
Backing Up Large Files..Or A Large Amount Of Files Scott D. Weber For Unuathorized Thoughts Inc. Computer Support 1 09-19-2003 07:28 PM



Advertisments