Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C++ > Creating large arrays..

Reply
Thread Tools

Creating large arrays..

 
 
hamishd
Guest
Posts: n/a
 
      01-28-2008
What is the best way to store large arrays of numbers (eg, 4-byte
integers)?

Say I want to store an array of 1billion 4-byte integers.. If my
computer has > 4gB memory, then is this possible?

int Large_Array[1000000000];

This will cause a "stack overflow" when i try and execute. How can I
do this?
 
Reply With Quote
 
 
 
 
Alf P. Steinbach
Guest
Posts: n/a
 
      01-28-2008
* hamishd:
> What is the best way to store large arrays of numbers (eg, 4-byte
> integers)?
>
> Say I want to store an array of 1billion 4-byte integers.. If my
> computer has > 4gB memory, then is this possible?


Depends on the computer and C++ implementation.


> int Large_Array[1000000000];
>
> This will cause a "stack overflow" when i try and execute. How can I
> do this?


You might try to allocate it dynamically, by using

std::vector<int> Large_Array( 1000000000 );

However, if you're skirting the limits of your computer's memory
capacity, then it's not a good idea even if it seems to work.

You'd be better off using some disk-based structure, processing only
parts of it at a time.


Cheers, & hth.,

- Alf

--
A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
 
Reply With Quote
 
 
 
 
hamishd
Guest
Posts: n/a
 
      01-28-2008
On Jan 28, 11:58*am, "Alf P. Steinbach" <(E-Mail Removed)> wrote:
> * hamishd:
>
> > What is the best way to store large arrays of numbers (eg, 4-byte
> > integers)?

>
> > Say I want to store an array of 1billion 4-byte integers.. If my
> > computer has > 4gB memory, then is this possible?

>
> Depends on the computer and C++ implementation.
>
> > int Large_Array[1000000000];

>
> > This will cause a "stack overflow" when i try and execute. How can I
> > do this?

>
> You might try to allocate it dynamically, by using
>
> * *std::vector<int> Large_Array( 1000000000 );
>
> However, if you're skirting the limits of your computer's memory
> capacity, then it's not a good idea even if it seems to work.


thanks.

> You'd be better off using some disk-based structure, processing only
> parts of it at a time.


What are the quicker ways of doing this?
 
Reply With Quote
 
Alf P. Steinbach
Guest
Posts: n/a
 
      01-28-2008
* hamishd:
> On Jan 28, 11:58 am, "Alf P. Steinbach" <(E-Mail Removed)> wrote:
>> * hamishd:
>>
>>> What is the best way to store large arrays of numbers (eg, 4-byte
>>> integers)?
>>> Say I want to store an array of 1billion 4-byte integers.. If my
>>> computer has > 4gB memory, then is this possible?

>> Depends on the computer and C++ implementation.
>>
>>> int Large_Array[1000000000];
>>> This will cause a "stack overflow" when i try and execute. How can I
>>> do this?

>> You might try to allocate it dynamically, by using
>>
>> std::vector<int> Large_Array( 1000000000 );
>>
>> However, if you're skirting the limits of your computer's memory
>> capacity, then it's not a good idea even if it seems to work.

>
> thanks.
>
>> You'd be better off using some disk-based structure, processing only
>> parts of it at a time.

>
> What are the quicker ways of doing this?


Depends on what you're doing.

E.g. it might be that what you're doing best fits some kind of tree
structure (note: a B-tree isn't a binary tree ).

Or it might be that what you're really doing is just some kind of
sequential processing, in which case read in suitably large chunks of a
file at a time, process them, and write them back or to some other file.

If it weren't for the size problem a memory mapped file would be ideal,
and IIRC the Boost library provides some fairly portable support for that.

However, the size problem means resorting to old-fashioned reads and
writes, and the clue there for efficiency is to do large enough chunks,
and perhaps also preferentially going down to at least the C file i/o
level instead of C++ iostreams (but it might be good idea to measure!).


Cheers, & hth.,

- Alf

--
A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
 
Reply With Quote
 
tOmMy
Guest
Posts: n/a
 
      01-28-2008
why not use pointer instead of array?


On 1月28日, 下午7时38分, hamishd <(E-Mail Removed)> wrote:
> What is the best way to store large arrays of numbers (eg, 4-byte
> integers)?
>
> Say I want to store an array of 1billion 4-byte integers.. If my
> computer has > 4gB memory, then is this possible?
>
> int Large_Array[1000000000];
>
> This will cause a "stack overflow" when i try and execute. How can I
> do this?


 
Reply With Quote
 
hamishd
Guest
Posts: n/a
 
      01-28-2008
On Jan 28, 3:13*pm, "Alf P. Steinbach" <(E-Mail Removed)> wrote:
> * hamishd:
>
>
>
>
>
> > On Jan 28, 11:58 am, "Alf P. Steinbach" <(E-Mail Removed)> wrote:
> >> * hamishd:

>
> >>> What is the best way to store large arrays of numbers (eg, 4-byte
> >>> integers)?
> >>> Say I want to store an array of 1billion 4-byte integers.. If my
> >>> computer has > 4gB memory, then is this possible?
> >> Depends on the computer and C++ implementation.

>
> >>> int Large_Array[1000000000];
> >>> This will cause a "stack overflow" when i try and execute. How can I
> >>> do this?
> >> You might try to allocate it dynamically, by using

>
> >> * *std::vector<int> Large_Array( 1000000000 );

>
> >> However, if you're skirting the limits of your computer's memory
> >> capacity, then it's not a good idea even if it seems to work.

>
> > thanks.

>
> >> You'd be better off using some disk-based structure, processing only
> >> parts of it at a time.

>
> > What are the quicker ways of doing this?

>
> Depends on what you're doing.


I'm storing the frequency that a value occurs (value ranging from 1..
a billion).

However.. I suspect the array maybe quite sparce. That is, not all
values will occur at all, so there may be a large number of zeroes.


 
Reply With Quote
 
Daniel Kraft
Guest
Posts: n/a
 
      01-28-2008
hamishd wrote:
>>>>> What is the best way to store large arrays of numbers (eg, 4-byte
>>>>> integers)?
>>>>> Say I want to store an array of 1billion 4-byte integers.. If my
>>>>> computer has > 4gB memory, then is this possible?
>>>> Depends on the computer and C++ implementation.
>>>>> int Large_Array[1000000000];
>>>>> This will cause a "stack overflow" when i try and execute. How can I
>>>>> do this?
>>>> You might try to allocate it dynamically, by using
>>>> std::vector<int> Large_Array( 1000000000 );
>>>> However, if you're skirting the limits of your computer's memory
>>>> capacity, then it's not a good idea even if it seems to work.
>>> thanks.
>>>> You'd be better off using some disk-based structure, processing only
>>>> parts of it at a time.
>>> What are the quicker ways of doing this?

>> Depends on what you're doing.

>
> I'm storing the frequency that a value occurs (value ranging from 1..
> a billion).
>
> However.. I suspect the array maybe quite sparce. That is, not all
> values will occur at all, so there may be a large number of zeroes.


You could use either a hash-table or even std::map<int, int> for this
job, I think.

Daniel

--
Got two Dear-Daniel-Instant Messages
by MSN, associate ICQ with stress--so
please use good, old E-MAIL!
 
Reply With Quote
 
Grizlyk
Guest
Posts: n/a
 
      01-29-2008
hamishd <(E-Mail Removed)> wrote:
>
> What is the best way to store large arrays of numbers
> ...
> If my computer has > 4gB memory
> ...
> int Large_Array[1000000000];
> This will cause a "stack overflow" when i try and execute.


Do you want to write portable application? C++ knows nothing about
installed memory of your machine. There are some layers between C++
and hardware, you need to refer to the layers to make portable
implementation of the array.

Do you want to write concrete hardware application? You need to refer
to the concrete hardware. For some executable environment more than 32
bit address RAM can be allowed for you and you will not get "stack
overflow" there.

For example, you can process the array with own wrapper (user defined
class) even for very low-memory executable environment or with limited
or poor virtual memory management, implementing the large array
storage as binary file with random access to manage memory explicitly.
The own wrapper also can use own types to represent indexes of the
array (to remove limits of xxx_MAX).

I think, long arrays are known for many years, so must be well-known
wrappers for the work.

Maksim A. Polyanin
http://grizlyk1.narod.ru/cpp_new
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
median of large data set (from large file) friend.05@gmail.com Perl Misc 5 04-02-2009 04:06 AM
loop beats generator expr creating large dict!? George Young Python 7 10-04-2006 04:30 PM
[Urgent] Is there a size limit on returning a large dataset or a large typed array from web service? Ketchup ASP .Net Web Services 1 05-25-2004 10:11 AM
Backing Up Large Files..Or A Large Amount Of Files Scott D. Weber For Unuathorized Thoughts Inc. Computer Support 1 09-19-2003 07:28 PM
Creating a large number of charts automatically Alan ASP General 1 09-01-2003 10:22 PM



Advertisments