Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C++ > Array Size - Upper Limit

Reply
Thread Tools

Array Size - Upper Limit

 
 
D. Susman
Guest
Posts: n/a
 
      12-18-2007
Hi,

I know that this issue is indeed strictly operating system dependent
but I am just curious:

I have a five dimensional array, whose size sums to 68 MB (almost).
This array is contained by class X. When I simply attempt to create an
instance of that class, I get segmentation fault.

I am working on Solaris (which has 8 KB pages ). What may the case be?
There seems to be an upper limit for the size of an array.

Thanks.
 
Reply With Quote
 
 
 
 
Victor Bazarov
Guest
Posts: n/a
 
      12-18-2007
D. Susman wrote:
> I know that this issue is indeed strictly operating system dependent
> but I am just curious:
>
> I have a five dimensional array, whose size sums to 68 MB (almost).
> This array is contained by class X. When I simply attempt to create an
> instance of that class, I get segmentation fault.


What does "simply attempt to create an instance" mean? An automatic
object? Have you tried creating it dynamically?

> I am working on Solaris (which has 8 KB pages ). What may the case be?


If that's an automatic object you're trying to create, your stack is
not big enough to contain it. Try increasing the stack size (usually
it is a linker option, RTFM to learn which one it's on your system).

Also, consider that putting a 68 MB stress on your stack may not be
a good idea.

> There seems to be an upper limit for the size of an array.


.... but I am not certain you've reached it yet. There are other,
more severe limitations on your program, and they can be at play
here.

V
--
Please remove capital 'A's when replying by e-mail
I do not respond to top-posted replies, please don't ask


 
Reply With Quote
 
 
 
 
Jim Langston
Guest
Posts: n/a
 
      12-18-2007
D. Susman wrote:
> Hi,
>
> I know that this issue is indeed strictly operating system dependent
> but I am just curious:
>
> I have a five dimensional array, whose size sums to 68 MB (almost).
> This array is contained by class X. When I simply attempt to create an
> instance of that class, I get segmentation fault.


A stack seems to always be smaller than a free store. There are usually
ways when compiling a program to increase the stack size (see your compiler
documentation) but 68MB is probably pushing it. As Victor said, you should
probably move the allocation of the memory to the freestore, which generally
has a much larger limit. Use new and some sort of smart pointer (I
personally would use a naked pointer, but that is generally frouwned upon).

> I am working on Solaris (which has 8 KB pages ). What may the case be?
> There seems to be an upper limit for the size of an array.


Check your compiler documentation for increasing the size of the stack.
That documentation should state the default size of the stack and how large
it can be increased to with some compiler switch.

--
Jim Langston
http://www.velocityreviews.com/forums/(E-Mail Removed)


 
Reply With Quote
 
James Kanze
Guest
Posts: n/a
 
      12-19-2007
On Dec 18, 5:05 pm, "Victor Bazarov" <(E-Mail Removed)> wrote:
> D. Susman wrote:
> > I know that this issue is indeed strictly operating system dependent
> > but I am just curious:


> > I have a five dimensional array, whose size sums to 68 MB (almost).
> > This array is contained by class X. When I simply attempt to create an
> > instance of that class, I get segmentation fault.


> What does "simply attempt to create an instance" mean? An automatic
> object? Have you tried creating it dynamically?


Or statically? Or throwing it as an exception?

> > I am working on Solaris (which has 8 KB pages ). What may the case be?


> If that's an automatic object you're trying to create, your stack is
> not big enough to contain it. Try increasing the stack size (usually
> it is a linker option, RTFM to learn which one it's on your system).


Usually, it's a simple command which you execute before invoking
the program. Under Solaris, "ulimit -s unlimited" should work.
(With ulimit -s set to unlimited, you can get as much stack as
you could get with dynamic allocation.)

> Also, consider that putting a 68 MB stress on your stack may
> not be a good idea.


Doing it once shouldn't cause any problems on a modern machine
(32 bits or more). I wouldn't suggest doing it in a lot of
different functions, which might end up calling each other, but
if there's only one instance for the life of the program,
declaring it in main is probably all right. (With a data
structure of this size, he's not going to be porting to a 16 bit
machine anyway.)

> > There seems to be an upper limit for the size of an array.


> ... but I am not certain you've reached it yet.


The formal upper limit is numeric_limits<size_t>::max(). Under
Solaris, either 4E09 or 1.8E19, depending on compiler options.
In practice, of course, the program will have to allocate memory
for it at some time, and that's likely to fail long before you
reach the formal limit (especially in the second case).

> There are other, more severe limitations on your program, and
> they can be at play here.


Yes. Note that even if the system accepts the allocation, if
you start accessing it randomly, cache misses will slow the
program down considerably. For even larger allocations, where
there isn't enough physical memory, page faults can slow the
system down to the point of becoming unusable. (When I test our
application here under Purify---which increases the memory
footprint significantly---, I have to terminate every other
application on the system. Otherwise, X gets paged out, and the
disk starts thrashing every time I move the cursor.)

--
James Kanze (GABI Software) email:(E-Mail Removed)
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
 
Reply With Quote
 
James Kanze
Guest
Posts: n/a
 
      12-19-2007
On Dec 18, 6:29 pm, "Jim Langston" <(E-Mail Removed)> wrote:
> D. Susman wrote:
> > I know that this issue is indeed strictly operating system dependent
> > but I am just curious:


> > I have a five dimensional array, whose size sums to 68 MB (almost).
> > This array is contained by class X. When I simply attempt to create an
> > instance of that class, I get segmentation fault.


> A stack seems to always be smaller than a free store.


On the systems I use regularly (Solaris on Sparc and Linux on a
PC), the stack and the free store compete for the same memory,
with the first to ask for it getting it. (Once allocated for
one, the memory can never be used for the other.)

By default, an artificial limit is imposed by the OS on the
stack size, so that a program with endless recursion will
terminate in my lifetime. The user can normally specify this
limit (8 Meg by default on my Sparc), however, or specify that
he doesn't want it at all.

Of course, if the stack becomes so big that you start paging on
every function call, it's going to slow things down a bit.
(Function calls don't actually require any memory accesses on a
Sparc, but it would be a rare function that didn't actually make
some accesses somewhere, at least if the code isn't optimized.)

> There are usually ways when compiling a program to increase
> the stack size (see your compiler documentation) but 68MB is
> probably pushing it.


One, you don't have to recompile, just modify the runtime
environment, and two, on a modern machine (at least on a PC or a
Sparc), there's absolutely no problem with 68MB on the stack if
you've got enough memory. (On one of the Linux machines here,
with a fair amount of memory, I was able to recurse 30 times
with a data structure of 68 Meg on the stack. On my Sparc, with
a lot less real memory, I interrupted the process after 16
recursions, because the disk thrashing was interfering with
other more important processes.)

> As Victor said, you should probably move the allocation of the
> memory to the freestore, which generally has a much larger
> limit. Use new and some sort of smart pointer (I personally
> would use a naked pointer, but that is generally frouwned
> upon).


It depends on where and what. If he's only allocating a single
instance, either a static variable or an automatic variable in
main would do the job just as well. (I'd go for the static
instance, since this would mean that the program could run in
the default environment, without the user having to invoke
ulimit first.) If he needs several, at least under Solaris, it
really doesn't matter whether he uses dynamic allocation or auto
variables, as far as memory goes. I'd go with dynamic
allocation, however, because it allows error checking---you're
informed if the allocation fails, rather than just core dumping.

> > I am working on Solaris (which has 8 KB pages ). What may
> > the case be? There seems to be an upper limit for the size
> > of an array.


> Check your compiler documentation for increasing the size of
> the stack. That documentation should state the default size
> of the stack and how large it can be increased to with some
> compiler switch.


"man ulimit" should be sufficient.

--
James Kanze (GABI Software) email:(E-Mail Removed)
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Emacs, VHDL Mode - Upper Case Enum Values & Upper Case Constants (not working)? BlackHelicopter VHDL 0 01-31-2013 04:12 AM
What upper limit for megapixels? SS Digital Photography 54 12-12-2007 07:59 AM
upper limit - no of fstreams Gaurav Sharma C++ 1 11-02-2004 01:49 AM
upper memory limit =?iso-8859-1?q?Leandro?= Python 4 08-10-2004 05:22 AM
Upper limit for Megapixels? Morgan Ohlson Digital Photography 47 02-20-2004 10:54 PM



Advertisments