Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > Handling large amounts of data

Reply
Thread Tools

Handling large amounts of data

 
 
Wayne Marsh
Guest
Posts: n/a
 
      11-20-2004
Hi all.

I am working on an audio application which needs reasonably fast access to
large amounts of data. For example, the program may load a 120 second
stereo sound sample stored at 4bytes per sample, which would mean over
40MB of data at a 44100Hz sampling rate.

Now, what would be a good way to handle all of this data? Ideally, for the
sake of my own sanity and the algorithms within directly functional
portions of the code, I'd like to interface with the data via normal array
syntax. Are arrays of this size really suitable, or would there be a
better way? Writing the data to disk and then memory mapping the files
seemed like an option, although I suspect that would be analogous to the
operating system's virtual memory system.

Any ideas?
 
Reply With Quote
 
 
 
 
jacob navia
Guest
Posts: n/a
 
      11-20-2004
Wayne Marsh wrote:
> Hi all.
>
> I am working on an audio application which needs reasonably fast access
> to large amounts of data. For example, the program may load a 120 second
> stereo sound sample stored at 4bytes per sample, which would mean over
> 40MB of data at a 44100Hz sampling rate.
>
> Now, what would be a good way to handle all of this data? Ideally, for
> the sake of my own sanity and the algorithms within directly functional
> portions of the code, I'd like to interface with the data via normal
> array syntax. Are arrays of this size really suitable, or would there be
> a better way? Writing the data to disk and then memory mapping the files
> seemed like an option, although I suspect that would be analogous to the
> operating system's virtual memory system.
>
> Any ideas?


It depends on how much memory is there in the computer. If you work
in a PC environment, machines now come routinely equipped with 1GB
of RAM, and 40MB is nothing. Just load it into RAM and use it as an
array. The VM system will do the paging for you if your OS is UNIX
or windows.

It would be surprising if you wanted to process all this data in an
embedded system with a few K of RAM anyway.

jacob
 
Reply With Quote
 
 
 
 
Mac
Guest
Posts: n/a
 
      11-20-2004
On Sat, 20 Nov 2004 16:05:00 +0000, Wayne Marsh wrote:

> Hi all.
>
> I am working on an audio application which needs reasonably fast access to
> large amounts of data. For example, the program may load a 120 second
> stereo sound sample stored at 4bytes per sample, which would mean over
> 40MB of data at a 44100Hz sampling rate.
>
> Now, what would be a good way to handle all of this data? Ideally, for the
> sake of my own sanity and the algorithms within directly functional
> portions of the code, I'd like to interface with the data via normal array
> syntax. Are arrays of this size really suitable, or would there be a
> better way? Writing the data to disk and then memory mapping the files
> seemed like an option, although I suspect that would be analogous to the
> operating system's virtual memory system.
>
> Any ideas?


On a PC or similar, 40 MB isn't really that much data nowadays. I would
start by just reading the whole file into an array and accessing the data
via the array.

If that approach proves problematic, you can go back and try to use memory
mapping (which isn't really on topic here) or what have you. This should
have little effect on the rest of your program, so it's not like you lose
anything if you have to change to the memory mapped approach.

If you are not envisioning a PC-type environment, then that is another
story.

--Mac

 
Reply With Quote
 
Malcolm
Guest
Posts: n/a
 
      11-20-2004

"Wayne Marsh" <(E-Mail Removed)> wrote
> I am working on an audio application which needs reasonably fast access to
> large amounts of data. For example, the program may load a 120 second
> stereo sound sample stored at 4bytes per sample, which would mean over
> 40MB of data at a 44100Hz sampling rate.
>
> Now, what would be a good way to handle all of this data?
>

There isn't really an answer. It depends on the exact platform, and nature
of calculations performed.
Some computers will happily chomp 40MB of data and represent it as a flat
array, whilst others will struggle. Often there is no point trying to
implement a virtual memory system of your own if the OS will do it for you.
In other cases there is a point. For instance if you need to access data in
5K chunks one megabyte apart, then a clever system of memory allocation will
beat the standard swap space algorithm hands down. However your data is
audio, so it is unlikely you want to do this.
The question is, do you really need random access over the whole 40MB array,
or can you treat the data as streamed? If you can treat it as streamed, then
it is probably best not to waste all that memory, unless you know that the
computer you are running on has the capacity to handle it, and the memory
would otherwise simply go unused.


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Opinion: Optimal way of holding large amounts of data between pages? David ASP .Net 0 06-21-2006 08:08 PM
HTTP POST to send large amounts of data? Bint HTML 1 03-19-2006 02:11 PM
transferring large amounts of data Andersen Java 2 11-03-2005 03:16 AM
What is the best email program for handling large amounts of incoming mail that needs to be filtered? C. B. Computer Support 19 04-13-2004 11:03 PM
Using Dataset for large amounts of data Brent ASP .Net 3 04-07-2004 01:43 PM



Advertisments