Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   C++ (http://www.velocityreviews.com/forums/f39-c.html)
-   -   basic memory question (http://www.velocityreviews.com/forums/t278282-basic-memory-question.html)

solartimba 10-09-2003 06:45 PM

basic memory question
 
I am a stats man that uses C++ to analyze stock data, but I have run
into a problem regarding memory usage.

First, though, I need to give a little background info. I want to
write a program that will open up 132 files containing daily stock
market data. I estimate that each file contains around 91,000 bytes.
Multiplied by 132, this equals 12 million bytes. Given that I have
512 megs of memory, I thought that I would have no problem having all
this data in memory at the same time. But just to be sure before I
wrote all the code, I did a simple test of creating a very large array
to find out what size would cause the system to crash. The test
program's code is below:


int main()
{
float array[259100]={4};
array[258999]=2222;
cout << "\nARRAY 1 " << array[258999];

return 0;

} //end main()

This program crashes if the array size is increased beyond 259,000.
Why? The array only takes up 1.04 million bytes.

So, my question is: what do I need to do to have all that data in
memory at the same time?

Thanks

Ivan Vecerina 10-09-2003 07:01 PM

Re: basic memory question
 
"solartimba" <kaferro@hotmail.com> wrote in message
news:1a7f7e51.0310091045.2784ea22@posting.google.c om...
> I am a stats man that uses C++ to analyze stock data, but I have run
> into a problem regarding memory usage.

....
> But just to be sure before I
> wrote all the code, I did a simple test of creating a very large array
> to find out what size would cause the system to crash. The test
> program's code is below:

....
> int main()
> {
> float array[259100]={4};
> array[258999]=2222;
> cout << "\nARRAY 1 " << array[258999];
>
> return 0;
>
> } //end main()


This function allocates the array as a local variable
of a fixed size (which typically means it is stored on
the stack if you know waht this is).
Local variables typically cannot be as large as
arrays allocated dynamically (on the heap) using
'new' or a standard container.

> This program crashes if the array size is increased beyond 259,000.
> Why? The array only takes up 1.04 million bytes.
>
> So, my question is: what do I need to do to have all that data in
> memory at the same time?


Use a container, such as std::vector:

#include <vector>
#include <iostream>

int main()
{
std::vector<float> array;
array.resize(1000000); // will just work
array[258999] = 2222;
std::cout << "\nARRAY 1 " << array[258999];
return 0;
}

If you find this code hard to understand,
I would recommend the book "Accelerated C++".
It should help for the type of thing you
are doing...


I hope this helps,
Ivan
--
http://ivan.vecerina.com
http://www.brainbench.com <> Brainbench MVP for C++



Unforgiven 10-09-2003 07:43 PM

Re: basic memory question
 
solartimba wrote:
> I am a stats man that uses C++ to analyze stock data, but I have run
> into a problem regarding memory usage.
>
> First, though, I need to give a little background info. I want to
> write a program that will open up 132 files containing daily stock
> market data. I estimate that each file contains around 91,000 bytes.
> Multiplied by 132, this equals 12 million bytes. Given that I have
> 512 megs of memory, I thought that I would have no problem having all
> this data in memory at the same time. But just to be sure before I
> wrote all the code, I did a simple test of creating a very large array
> to find out what size would cause the system to crash. The test
> program's code is below:
>
>
> int main()
> {
> float array[259100]={4};
> array[258999]=2222;


You are allocating arrays on the stack. Stack space for a program is
(usually) limited by the compiler. You must either increase the stack size
(look at your compiler's documentation for that) or allocate the array on
the heap. The best and safest way to do that is to use std::vector.

--
Unforgiven

A: Top Posting!
Q: What is the most annoying thing on Usenet?


klaas 10-09-2003 07:54 PM

Re: basic memory question
 
solartimba wrote:

> I am a stats man that uses C++ to analyze stock data, but I have run
> into a problem regarding memory usage.
>
> First, though, I need to give a little background info. I want to
> write a program that will open up 132 files containing daily stock
> market data. I estimate that each file contains around 91,000 bytes.
> Multiplied by 132, this equals 12 million bytes. Given that I have
> 512 megs of memory, I thought that I would have no problem having all
> this data in memory at the same time. But just to be sure before I
> wrote all the code, I did a simple test of creating a very large array
> to find out what size would cause the system to crash. The test
> program's code is below:
>
>

well probably ypour floating point number does not fit in a byte right?
probaly it fits in a word of four bytes. If you then multiply by four
you'll get the right amount right?


Jerry Coffin 10-10-2003 04:15 AM

Re: basic memory question
 
In article <1a7f7e51.0310091045.2784ea22@posting.google.com >,
kaferro@hotmail.com says...
> I am a stats man that uses C++ to analyze stock data, but I have run
> into a problem regarding memory usage.
>
> First, though, I need to give a little background info. I want to
> write a program that will open up 132 files containing daily stock
> market data. I estimate that each file contains around 91,000 bytes.
> Multiplied by 132, this equals 12 million bytes. Given that I have
> 512 megs of memory, I thought that I would have no problem having all
> this data in memory at the same time. But just to be sure before I
> wrote all the code, I did a simple test of creating a very large array
> to find out what size would cause the system to crash.


First of all, you almost certainly want to use memory mapping to handle
this. It's theoretically not portable, but nearly every reasonably
modern OS supports it.

The problem you're running into is that many systems limit the stack
size to (considerably) less than all of memory. There are a couple of
ways of handling this: you can allocate the space statically or on the
heap, for a couple of the most obvious methods -- The easiest way is
probably to use std::vector instead of an array, in which case the space
will be allocated on the heap automatically.

--
Later,
Jerry.

The universe is a figment of its own imagination.


All times are GMT. The time now is 02:52 AM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.