«

Hi!

I'm seeking some answers about what seems to be a memory leak.

I have a loop that looks much like this:
double *largeArray = (double*) calloc();
for (...) {
printf("iteration #...\n");
for (...) {
double *foo = (double*) calloc();
.....
.....
largeArray[someIndex] = something;
free(foo);
}
}

Though the actual code is larger, it only differs in 20+ lines of
trivial math performed on stack variables.

Clearly, foo cannot be leaking since it's being freed (and no, it
cannot be allocated outside of the loop, since its size varies each
time.

Now, when I monitor memory usage with top it grows relatively quickly
(300K per pass over the outer loop), thus there ought to be a memory
leak. At first I thought that the "largeArray" was being optimized not
to calloc all at once, but rather on demand, page by page (which would
be bizzarre) but now I believe that might not be the case since the
"largeArray" is about 4000*4000 of double which should be about 16MB -
and I see usage of > 100MB after a few hundred iterations.

I'm using gcc 3.2.2 on i*86 Linux.
Any guesses would be appreciated.

Thanks!

Boris

In article < .com>,
<> wrote:
>"largeArray" is about 4000*4000 of double which should be about 16MB -

4000*4000 doubles is 16M * sizeof(double), which is 128MB if you have
8-byte doubles.

-- Richard

<> wrote in message
news: oups.com...
> Hi!
>
> I'm seeking some answers about what seems to be a memory leak.
>
> I have a loop that looks much like this:
> double *largeArray = (double*) calloc();
> for (...) {
> printf("iteration #...\n");
> for (...) {
> double *foo = (double*) calloc();
> ....
> ....
> largeArray[someIndex] = something;
> free(foo);
> }
> }
>
> Though the actual code is larger, it only differs in 20+ lines of
> trivial math performed on stack variables.
>
> Clearly, foo cannot be leaking since it's being freed (and no, it
> cannot be allocated outside of the loop, since its size varies each
> time.
>
> Now, when I monitor memory usage with top it grows relatively quickly
> (300K per pass over the outer loop), thus there ought to be a memory
> leak. At first I thought that the "largeArray" was being optimized not
> to calloc all at once, but rather on demand, page by page (which would
> be bizzarre) but now I believe that might not be the case since the
> "largeArray" is about 4000*4000 of double which should be about 16MB -
> and I see usage of > 100MB after a few hundred iterations.
>
> I'm using gcc 3.2.2 on i*86 Linux.
> Any guesses would be appreciated.
>

I really do not see an issue with the code you have provided.
(Other than casting where unnecessary). It is too incomplete.
Can you not provide all of it? If not, I would recommend the
use of valgrind here. But that is off-topic for this newsgroup.



--
j

On 2005-01-31 12:18:44 -0500, said:

> Hi!
>
> I'm seeking some answers about what seems to be a memory leak.

[snip]

> ... since the
> "largeArray" is about 4000*4000 of double which should be about 16MB -
> and I see usage of > 100MB after a few hundred iterations.

Do the math again:

4,000 * 4,000
= 16,000,000

If sizeof(double) is 8 then:

8B * 16,000,000
= 128,000,000B
≈ 122 MB

So your usage of > 100MB seems to be right in line with what should be
expected.

--
Clark S. Cox, III

Right-O. I'm an idiot: >100MB is exactly the right space usage.
However, why is it not all allocated with the first calloc of
largeArray - why do I see 'top' report ever-growing usage? This is
where I would probably want to use -fprefetch-loop-arrays, which is not
supported on my architecture according to gcc

As for providing more code, I could - but the rest of it is just junk -
this is all of the relevant code.

Boris

wrote:
> Hi!
>
> I'm seeking some answers about what seems to be a memory leak.
>
> I have a loop that looks much like this:
> double *largeArray = (double*) calloc();
> for (...) {
> printf("iteration #...\n");
> for (...) {
> double *foo = (double*) calloc();
> ....
> ....
> largeArray[someIndex] = something;
> free(foo);
> }
> }
>
> Though the actual code is larger, it only differs in 20+ lines of
> trivial math performed on stack variables.
>
> Clearly, foo cannot be leaking since it's being freed (and no, it
> cannot be allocated outside of the loop, since its size varies each
> time.
>
> Now, when I monitor memory usage with top it grows relatively quickly
> (300K per pass over the outer loop), thus there ought to be a memory
> leak. At first I thought that the "largeArray" was being optimized
not
> to calloc all at once, but rather on demand, page by page (which
would
> be bizzarre) but now I believe that might not be the case since the
> "largeArray" is about 4000*4000 of double which should be about 16MB
-
> and I see usage of > 100MB after a few hundred iterations.
>
> I'm using gcc 3.2.2 on i*86 Linux.
> Any guesses would be appreciated.
>
> Thanks!
>
> Boris

wrote:
> Right-O. I'm an idiot: >100MB is exactly the right space usage.
> However, why is it not all allocated with the first calloc of
> largeArray - why do I see 'top' report ever-growing usage? This is
> where I would probably want to use -fprefetch-loop-arrays, which is not
> supported on my architecture according to gcc
>
> As for providing more code, I could - but the rest of it is just junk -
> this is all of the relevant code.

But it will not work when pasted into some sort of main() function.
>
> Boris
>
> wrote:
>
>>Hi!
>>
>>I'm seeking some answers about what seems to be a memory leak.
>>
>>I have a loop that looks much like this:
#include <stdlib.h>
#include <stdio.h>

int main (void)
{
>>double *largeArray = (double*) calloc();
how much are you calloc()ing.
>>for (...) {
where are you looping

>>printf("iteration #...\n");
>>for (...) {
dito

>>double *foo = (double*) calloc();
how much are you calloc()ing
>>....
>>....
>>largeArray[someIndex] = something;
where are someindex and something declared/initialized

>>free(foo);
>>}
>>}
where is largeArray free()d

return 0;
}

Now, give us that stuff requested or create a minimal example --
then we can help you.

Note that calloc() does not necessarily make sense for doubles

Please do not top-post.


Cheers
Michael
>>
>>Though the actual code is larger, it only differs in 20+ lines of
>>trivial math performed on stack variables.
>>
>>Clearly, foo cannot be leaking since it's being freed (and no, it
>>cannot be allocated outside of the loop, since its size varies each
>>time.
>>
>>Now, when I monitor memory usage with top it grows relatively quickly
>>(300K per pass over the outer loop), thus there ought to be a memory
>>leak. At first I thought that the "largeArray" was being optimized
>
> not
>
>>to calloc all at once, but rather on demand, page by page (which
>
> would
>
>>be bizzarre) but now I believe that might not be the case since the
>>"largeArray" is about 4000*4000 of double which should be about 16MB
>
> -
>
>>and I see usage of > 100MB after a few hundred iterations.
>>
>>I'm using gcc 3.2.2 on i*86 Linux.
>>Any guesses would be appreciated.
>>
>>Thanks!
>>
>>Boris
>
>


--
E-Mail: Mine is an /at/ gmx /dot/ de address.

writes:
> I'm seeking some answers about what seems to be a memory leak.
>
> I have a loop that looks much like this:
> double *largeArray = (double*) calloc();
> for (...) {
> printf("iteration #...\n");
> for (...) {
> double *foo = (double*) calloc();
> ....
> ....
> largeArray[someIndex] = something;
> free(foo);
> }
> }
>
> Though the actual code is larger, it only differs in 20+ lines of
> trivial math performed on stack variables.
>
> Clearly, foo cannot be leaking since it's being freed (and no, it
> cannot be allocated outside of the loop, since its size varies each
> time.
>
> Now, when I monitor memory usage with top it grows relatively quickly
> (300K per pass over the outer loop), thus there ought to be a memory
> leak. At first I thought that the "largeArray" was being optimized not
> to calloc all at once, but rather on demand, page by page (which would
> be bizzarre) but now I believe that might not be the case since the
> "largeArray" is about 4000*4000 of double which should be about 16MB -
> and I see usage of > 100MB after a few hundred iterations.

Apart from your miscalculation of the size allocated for largeArray,
there's no guarantee that free() gives memory back to the operating
system. Very likely it stays within your program and becomes
available for further allocation. You don't give us a clue about what
arguments you're giving to calloc(), but it's possible that you're
fragmenting the heap and making it difficult for the system to
re-allocate the memory you've freed.

I would probably add some printf() statements to log all the calls to
calloc() and free(). For example:

double *foo = calloc(something, something_else);
/*
* don't cast the result of malloc() or calloc().
*/
printf("foo = calloc("%lu, %lu) --> [%p]\n",
(unsigned long)something,
(unsigned long)something_else,
(void*)foo);
...
printf("free(foo), foo=[%p]\n", (void*)foo);
free(foo);

Analyze the results and make sure you're freeing everything you
allocate. If not, there's your problem; if so, the displayed
addresses may tell you something, or there may be some system-specific
way to trace the internal behavior of calloc() and free().

Incidentally, it's not safe to assume that calloc() will set all the
doubles in your allocated array to 0.0. It sets the allocated memory
to all-bits-zero. This is often the representation of 0.0, but the
language doesn't guarantee it.

--
Keith Thompson (The_Other_Keith) kst- <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.

wrote:
> Right-O. I'm an idiot: >100MB is exactly the right space usage.
> However, why is it not all allocated with the first calloc of
> largeArray - why do I see 'top' report ever-growing usage? This is
> where I would probably want to use -fprefetch-loop-arrays, which is
not
> supported on my architecture according to gcc

Probably your operating system is doing 'lazy allocation'. It will
allocate you an address space but not actually claim that memory yet.

Then when you try and access memory in the space you have been
given, it will go and actually allocate that memory.

If there is not actually any memory available then it will die
in a screaming heap, or start swapping endlessly.

I think the point of lazy allocation is so that if a programmer
is lazy and just mallocs a huge chunk at the start, then other
applications do not need to suffer the effects of having not
much memory available.

"Old Wolf" <> writes:
> wrote:
>> Right-O. I'm an idiot: >100MB is exactly the right space usage.
>> However, why is it not all allocated with the first calloc of
>> largeArray - why do I see 'top' report ever-growing usage? This is
>> where I would probably want to use -fprefetch-loop-arrays, which is
>> not supported on my architecture according to gcc
>
> Probably your operating system is doing 'lazy allocation'. It will
> allocate you an address space but not actually claim that memory yet.
>
> Then when you try and access memory in the space you have been
> given, it will go and actually allocate that memory.
>
> If there is not actually any memory available then it will die
> in a screaming heap, or start swapping endlessly.
>
> I think the point of lazy allocation is so that if a programmer
> is lazy and just mallocs a huge chunk at the start, then other
> applications do not need to suffer the effects of having not
> much memory available.

It's pretty clear that lazy allocation is non-conforming. A program
should be able to determine whether enough memory is available when it
attempts to allocate it; that's why malloc() provides a simple and
clear mechanism for reporting failure. There's no way a program can
fail gracefully if the OS randomly kills it when it tries to access
memory it thinks it's already allocated.

The OP was using calloc(), which zeros the allocated memory, but
perhaps the system simulates that (so that the memory which springs
into existence when it's accessed looks like it's already filled with
zeros).

If your system does lazy allocation, one way to make it act as if it
were more nearly conforming would be to fill the allocated memory
with, say, 0xff bytes immediately after allocating it. That still
won't let it fail gracefully, but at least the failure will occur
sooner rather than later.

An experiment the OP might try is to fill the allocated memory with
some non-zero value immediately after calloc(), then fill it with
zeros again. Obviously this is going to slow things down (so you
won't want to do this in your production version), but it could be
useful to see whether this affects the memory behavior.

--
Keith Thompson (The_Other_Keith) kst- <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.

In article <>,
Keith Thompson <kst-> wrote:

> An experiment the OP might try is to fill the allocated memory with
> some non-zero value immediately after calloc(), then fill it with
> zeros again. Obviously this is going to slow things down (so you
> won't want to do this in your production version), but it could be
> useful to see whether this affects the memory behavior.

I have seen exactly this method being used in serious production code -
a function "my_malloc ()" with the same arguments as malloc, that would
call malloc (), install a signal handler, fill the malloc ()'d pointer
with some data, and finally return the pointer. If anything went wrong
while filling the allocated memory, the signal handler would stop the
signal from propagating; in that case the pointer was free()d and the
function returned NULL. Truly horrible code to attempt to get a
conforming implementation.