Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   C Programming (http://www.velocityreviews.com/forums/f42-c-programming.html)
-   -   total array size limitations? (http://www.velocityreviews.com/forums/t675530-total-array-size-limitations.html)

glennklockwood 03-13-2009 09:19 PM

total array size limitations?
 
I'm in the process of porting some FORTRAN code to C and am running
into a problem where the program crashes if I set the size of arrays
too large. In the below code, my program runs fine if I set MAXN to
50000 or less, but at 60000 or more, the program immediately crashes
and GDB indicates the error is at the printf line.

What's strange is that I've tried this by separating my 2-dimensional
arrays into a bunch of one-dimensional arrays (eg, xc[60000][6] into
xc0[60000], xc1[60000], xc2[60000], etc) and the same sort of crashing
stopped if I commented out a few of those one-dimensional
declarations. Because of this, I know that the problem isn't with one
giant array, but the total size of the arrays I'm declaring. I assume
this all has to do with some sort of memory limit and a lot of spotty
advice posted over the internet in similar cases have suggested it has
something to do with a 64K stack size, but this doesn't make much
sense to me.

Can anyone shine some light on what exactly is going wrong and how I
can avoid it? By my calculation the size of these declared variables
is a little over 9MB which isn't a huge amount, and the FORTRAN
version of this code works for MAXN in excess of 100000.

If it matters, I am compiling on linux amd64, and this problem has
happened with both GCC 4.1 and Sun's C compiler (SS12).

Thanks much.

glenn

-- Begin code --

#define MAXN 60000
#define MAXLT 10

int main()
{
double ener[100][MAXLT], ihite[100][MAXLT];
double xc[MAXN][6], yc[MAXN][6], zc[MAXN][6];
double aij[MAXLT][MAXLT], bij[MAXLT][MAXLT], amass[MAXLT],
twopi[MAXLT][MAXLT], eta[MAXLT][MAXLT], ze[MAXLT][MAXLT],
rhop[MAXLT][MAXLT], za[MAXLT];
int ltype[MAXN], ibx[MAXN], iby[MAXN], ibz[MAXN];
char k9[16];

double etot, xl, yl, zl, zreft, deltim, temp;
int nmol, istill, nsur, jread, icalc, nbin;

int i,j;
int dblesize, intsize;
FORTRAN_OFFSET_TYPE offset;
FORTRAN_OFFSET_TYPE recsize;
FILE *fp;

int inum[MAXLT], iff;

recsize = 0;
dblesize = sizeof(xl);
intsize = sizeof(i);

printf( " What file?\n" );

.... (it crashes before the printf)

Kenny McCormack 03-13-2009 09:21 PM

Re: total array size limitations?
 
In article <55db1c79-87bf-4bbe-85d3-8239abfc5fdc@13g2000yql.googlegroups.com>,
glennklockwood <glennklockwood@gmail.com> wrote:
>I'm in the process of porting some FORTRAN code to C and am running
>into a problem where the program crashes if I set the size of arrays
>too large. In the below code, my program runs fine if I set MAXN to
>50000 or less, but at 60000 or more, the program immediately crashes
>and GDB indicates the error is at the printf line.


Any question that starts out like this, has only one answer:

The (total) size of automatic objects in C is strictly limited (*).
Use globals and/or dynamic allocation (malloc and friends) if at all
possible.

(*) And don't even think of using the 's word' in this group.


Bartc 03-13-2009 09:32 PM

Re: total array size limitations?
 

"glennklockwood" <glennklockwood@gmail.com> wrote in message
news:55db1c79-87bf-4bbe-85d3-8239abfc5fdc@13g2000yql.googlegroups.com...
> I'm in the process of porting some FORTRAN code to C and am running
> into a problem where the program crashes if I set the size of arrays
> too large. In the below code, my program runs fine if I set MAXN to
> 50000 or less, but at 60000 or more, the program immediately crashes
> and GDB indicates the error is at the printf line.


> double ener[100][MAXLT], ihite[100][MAXLT];
> double xc[MAXN][6], yc[MAXN][6], zc[MAXN][6];
> double aij[MAXLT][MAXLT], bij[MAXLT][MAXLT], amass[MAXLT],
> twopi[MAXLT][MAXLT], eta[MAXLT][MAXLT], ze[MAXLT][MAXLT],
> rhop[MAXLT][MAXLT], za[MAXLT];
> int ltype[MAXN], ibx[MAXN], iby[MAXN], ibz[MAXN];


Try putting static in front of these big arrays. Or moving the whole lot to
just before the main function, if the names will not clash.

--
Bartc


glennklockwood 03-13-2009 09:33 PM

Re: total array size limitations?
 
On Mar 13, 5:21*pm, gaze...@shell.xmission.com (Kenny McCormack)
wrote:
> The (total) size of automatic objects in C is strictly limited (*).
> Use globals and/or dynamic allocation (malloc and friends) if at all
> possible.
>
> (*) And don't even think of using the 's word' in this group.


Making everything global certainly did the trick; thank you for the
very quick reply. I suppose I should have checked the books a little
more thoroughly.

Nate Eldredge 03-13-2009 09:51 PM

Re: total array size limitations?
 
glennklockwood <glennklockwood@gmail.com> writes:

> I'm in the process of porting some FORTRAN code to C and am running
> into a problem where the program crashes if I set the size of arrays
> too large. In the below code, my program runs fine if I set MAXN to
> 50000 or less, but at 60000 or more, the program immediately crashes
> and GDB indicates the error is at the printf line.
>
> What's strange is that I've tried this by separating my 2-dimensional
> arrays into a bunch of one-dimensional arrays (eg, xc[60000][6] into
> xc0[60000], xc1[60000], xc2[60000], etc) and the same sort of crashing
> stopped if I commented out a few of those one-dimensional
> declarations. Because of this, I know that the problem isn't with one
> giant array, but the total size of the arrays I'm declaring. I assume
> this all has to do with some sort of memory limit and a lot of spotty
> advice posted over the internet in similar cases have suggested it has
> something to do with a 64K stack size, but this doesn't make much
> sense to me.


On a typical system, a C program has a fixed-size region of memory set
aside for a stack. This is a very simple memory allocation scheme;
there is a single pointer (normally in a CPU register) marking the
"top". When a function gets called, it grabs some space
from the top of the stack, moving the pointer to indicate the new top,
and uses this space to store its local variables. When it returns, it
restores the pointer to its original value, thus allowing the space to
be reused later.

In your code, since the arrays you declare are local to the function
main(), they are allocated in this manner. But it appears that the
total size of your arrays is bigger than the total size of the stack
(typically a few megabytes). So trying to access those arrays runs off
the end of the stack and causes a crash.

The simplest way to fix it, in your case, would be to declare those
arrays with the `static' keyword. This causes them to be placed in a
different area of memory, which doesn't need to be reused like the
stack. It also means that multiple calls to the function will all
access the same copy of the arrays, rather than each one getting its own
copy, but that is irrelevant here since I presume you don't intend to
have main() recursively call itself. There is also the possibly
convenient side effect that it will cause all the arrays to be
initialized with zeros.

On Unix, you can query and set the default stack size in kbytes with
"ulimit -s" (from a Bourne shell) or "limit stacksize" (from a C shell)
or the rlimit() function (from within the program), so that would be
another possibility. A third option would be to allocate those arrays
dynamically using malloc(), but that is a little awkward (though
certainly possible) to do with multidimensional arrays.

> Can anyone shine some light on what exactly is going wrong and how I
> can avoid it? By my calculation the size of these declared variables
> is a little over 9MB which isn't a huge amount, and the FORTRAN
> version of this code works for MAXN in excess of 100000.


I make it more like 36 MB. The C `double' type occupies 8 bytes on
x86/amd64.

> If it matters, I am compiling on linux amd64, and this problem has
> happened with both GCC 4.1 and Sun's C compiler (SS12).


FWIW, on the Linux amd64 box I have access to, the stack is 10 MB by
default. Yours could be configured differently, of course.

> Thanks much.
>
> glenn
>
> -- Begin code --
>
> #define MAXN 60000
> #define MAXLT 10
>
> int main()


I'll just mention that modern C standards call for you to use "void" as
the argument list of a function that doesn't take any arguments. So
this should read

int main(void)

(Technically speaking, there seems to be some dispute as to whether
"int main()" is actually illegal or merely obsolescent.)

CBFalconer 03-14-2009 12:26 AM

Re: total array size limitations?
 
glennklockwood wrote:
>
> I'm in the process of porting some FORTRAN code to C and am running
> into a problem where the program crashes if I set the size of arrays
> too large. In the below code, my program runs fine if I set MAXN to
> 50000 or less, but at 60000 or more, the program immediately crashes
> and GDB indicates the error is at the printf line.


Automatic storage is generally assigned on a stack (if the system
uses a stack). Stack size is normally limited. I suggest using
pointers and mallocing the associated storage.

--
[mail]: Chuck F (cbfalconer at maineline dot net)
[page]: <http://cbfalconer.home.att.net>
Try the download section.



Antoninus Twink 03-14-2009 11:05 AM

Re: total array size limitations?
 
On 14 Mar 2009 at 3:23, Richard Harter wrote:
><cbfalconer@yahoo.com> wrote:
>>I suggest using pointers and mallocing the associated storage.

>
> And, of course, one should check out hashlib and nmalloc.


*snigger*

You forgot ggets(), the swiss army knife of C programming that solves
every problem!


CBFalconer 03-15-2009 12:05 AM

Re: total array size limitations?
 
Malcolm McLean wrote:
> "Eric Sosman" <Eric.Sosman@sun.com> wrote:
>
>> Fortran variables have what C would call "static" storage (or at
>> any rate they did years ago when I used FORTRAN). The arrays in
>> your C program use "automatic" storage, which is rather different.
>> On many systems, the amount of space available for "automatic"
>> variables is considerably less than is provided for other classes
>> of data.

>
> Fortran 77 is still alive and kicking. I have to use it myself,
> sometimes.
>
> All arguments are passed by reference, and all variables are static.
> So code can be very optimal - no hard to detect pointer aliasing
> problems.


And no routines are re-entrant, nor is recurrance allowed or
feasible. It is impossible to isolate variables for use in a
single function.

--
[mail]: Chuck F (cbfalconer at maineline dot net)
[page]: <http://cbfalconer.home.att.net>
Try the download section.



CBFalconer 03-15-2009 12:11 AM

Re: total array size limitations?
 
Richard Harter wrote:
> CBFalconer <cbfalconer@yahoo.com> wrote:
>> glennklockwood wrote:
>>
>>> I'm in the process of porting some FORTRAN code to C and am
>>> running into a problem where the program crashes if I set the
>>> size of arrays too large. In the below code, my program runs
>>> fine if I set MAXN to 50000 or less, but at 60000 or more, the
>>> program immediately crashes and GDB indicates the error is at
>>> the printf line.

>>
>> Automatic storage is generally assigned on a stack (if the
>> system uses a stack). Stack size is normally limited. I
>> suggest using pointers and mallocing the associated storage.

>
> And, of course, one should check out hashlib and nmalloc.


Why? I failed to see any application for them in the OPs request.
Please teach us your magnificent method of detecting requirements
in verbiage such as the above.

--
[mail]: Chuck F (cbfalconer at maineline dot net)
[page]: <http://cbfalconer.home.att.net>
Try the download section.



CBFalconer 03-16-2009 12:41 AM

Re: total array size limitations?
 
Richard Harter wrote:
> CBFalconer <cbfalconer@yahoo.com> wrote:
>> Richard Harter wrote:
>>

.... snip ...
>>
>>> And, of course, one should check out hashlib and nmalloc.

>>
>> Why? I failed to see any application for them in the OPs request.
>> Please teach us your magnificent method of detecting requirements
>> in verbiage such as the above.

>
> Surely you must be able to find something. You have so often been
> able to find reason for recommending them in the most unpromising
> of circumstances.


I am not responsible for your inability to detect suitable
applications. I, on the other hand, having designed those modules,
am aware of their abilities and can notice useful applications.

--
[mail]: Chuck F (cbfalconer at maineline dot net)
[page]: <http://cbfalconer.home.att.net>
Try the download section.




All times are GMT. The time now is 08:06 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.