«

Hi,

I have a script that uses up a lot of memory to process some data and
write it to a very large file.
This script is written in C. I've made sure that all the memory
allocated in the script is released before making a system() call to
compress the file and then exiting the script.
The attempt to compress failed because fork returned an -1 error
stating that there is insufficient memory available. It seems C keeps
the memory locked until it ends. Is there away to get C to release the
memory allocated before it exits?

Appreciate your help,
/M

Well,

show us some code - that will greatly increase your chance of getting
usable help.

> This script is written in C.

I am no language laywer; but calling a compiled program written in C a
"script" is at least unconventional.

> Is there away to get C to release the memory allocated before it exits?

Well certainly YES.

Joakim

On 4 June, 11:40, Joakim Hove <joakim.h...@gmail.com> wrote:
> Well,
>
> show us some code - that will greatly increase your chance of getting
> usable help.
>
> > This script is written in C.
>
> I am no language laywer; but calling a compiled program written in C a
> "script" is at least unconventional.
>
> > Is there away to get C to release the memory allocated before it exits?
>
> Well certainly YES.

not necessarily in C there isn't.

Joakim Hove wrote:

>> Is there away to get C to release the memory allocated before it exits?
>
> Well certainly YES.

Using what? `free` is under no obligation to hand back memory to the
OS, as far as I can tell.

--
"It is seldom good news." ~Crystal Ball~, /The Tough Guide to Fantasyland/

Hewlett-Packard Limited Cain Road, Bracknell, registered no:
registered office: Berks RG12 1HN 690597 England

On 4 Juni, 13:00, Chris Dollin <chris.dol...@hp.com> wrote:
> Joakim Hove wrote:
> >> Is there away to get C to release the memory allocated before it exits?
>
> > Well certainly YES.
>
> Using what? `free` is under no obligation to hand back memory to the
> OS, as far as I can tell.
>
> --
> "It is seldom good news." * * *~Crystal Ball~, /The Tough Guide to Fantasyland/
>
> Hewlett-Packard Limited * * Cain Road, Bracknell, * * * * * * * *registered no:
> registered office: * * * * *Berks RG12 1HN * * * * * * * * * * * 690597 England

The code/program is several about 2000 lines with several long
functions so I didn't want to post it all here.
Since the "problem" is more theoretical I thought I'd ask a general
question that Chris I think answered.
If free doesn't hand back memory to the OS, how can I force it to do
so?

I wrote a little program that does a system call from C and it works
(see below). When I insert this in my main C program just before
exiting main() and after freeing all allocated memory the system()
call returns -1 and the error "not enough memory". So I think that
when the memory is freed it's not returned to the OS before the
program ends. Is there a solution to this problem?
This file that I'm trying to zip is 504 MB so it needs a lot of
memory.


#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

int main(void) {

int status = -9999;

char cmd[] = "/bin/bzip2 /home/c_programming/trunk/complete/
ECMWF_NWP_20080910-0000-00.NC";

printf("Zipping file using bzip2 via system call...\n");
status = system(cmd);
printf("%d\n",status);

if(status != 0) {
perror("Error found!");
printf("errno = %d.\n", errno);
fprintf(stderr,"Failed to zip output file");
exit(EXIT_FAILURE);
}

return EXIT_SUCCESS;

}


Thanks,
/Marston

Marston wrote:

> On 4 Juni, 13:00, Chris Dollin <chris.dol...@hp.com> wrote:
>> Joakim Hove wrote:
>> >> Is there away to get C to release the memory allocated before it exits?
>>
>> > Well certainly YES.
>>
>> Using what? `free` is under no obligation to hand back memory to the
>> OS, as far as I can tell.

> The code/program is several about 2000 lines with several long
> functions so I didn't want to post it all here.
> Since the "problem" is more theoretical I thought I'd ask a general
> question that Chris I think answered.
> If free doesn't hand back memory to the OS, how can I force it to do
> so?

You can't. (It's /possible/ that some particular implemementation
would have a `freeEveryThingAndGiveItBackToTheOSRightNowBang()`
function, but you certainly can't count on it.)

> I wrote a little program that does a system call from C and it works
> (see below). When I insert this in my main C program just before
> exiting main() and after freeing all allocated memory the system()
> call returns -1 and the error "not enough memory". So I think that
> when the memory is freed it's not returned to the OS before the
> program ends. Is there a solution to this problem?

Yes, the solution is quite easy for this particular problem.

#whatever

int main(void)
{
system( "yourFileCreatingProgram" );
system( "yourBzippingCommandLine" );
return 0;
}

Now the UseMuchMemory program has almost certainly finished and gone
before you run bzip.

[Of course you could just write this as a two-line bash/ruby/perl/python
script, but that would be off-topic.]

--
"You're down as expendable. You don't get a weapon." /Dark Lord of Derkholm/

Hewlett-Packard Limited registered office: Cain Road, Bracknell,
registered no: 690597 England Berks RG12 1HN

"Marston" <> wrote in message
news:7066c309-e22f-4fa6-88ea-...
On 4 Juni, 13:00, Chris Dollin <chris.dol...@hp.com> wrote:
> Joakim Hove wrote:

> >> > Is there away to get C to release the memory allocated before it
> >> > exits?
>
> > > Well certainly YES.
>
> > Using what? `free` is under no obligation to hand back memory to the
> > OS, as far as I can tell.
>
> If free doesn't hand back memory to the OS, how can I force it to do
> so?

Since you like to call your program a script, and are using a
system() call to run the second part of the script, why not write
a REAL script, with the first part running your memory hog
program, and the second part running the compress command.

If indeed your problem is that the OS still sees the gigantic
chunk of memory as allocated to your program, when the
memory hog program in your script exits, THEN the memory
should be available to other programs, such as your compress
command...

---
William Ernest Reid

Marston wrote:
> On 4 Juni, 13:00, Chris Dollin <chris.dol...@hp.com> wrote:
>> Joakim Hove wrote:
>>>> Is there away to get C to release the memory allocated before it
>>>> exits?
>>
>>> Well certainly YES.
>>
>> Using what? `free` is under no obligation to hand back memory to the
>> OS, as far as I can tell.

> If free doesn't hand back memory to the OS, how can I force it to do
> so?

If your program uses lots of small, fragmented allocations then C may have
trouble in recovering memory in big enough blocks to be usefully returned to
the OS. (Maybe it just doesn't include the logic to identify the blocks, but
I'm guessing.)

If your C's free() works better with fewer, bigger allocations, then you can
possibly create your own allocator which works with large malloc() blocks.
Then you deallocate these large blocks at the end.

> This file that I'm trying to zip is 504 MB so it needs a lot of
> memory.

That's surprising, I thought Zip only really needed memory for it's tables,
and would not be constrained by available memory. Maybe you need a better
zip program.

--
Bart

On 4 June, 15:06, Richard <rgrd...@gmail.com> wrote:
> nick_keighley_nos...@hotmail.com writes:
> > On 4 June, 11:40, Joakim Hove <joakim.h...@gmail.com> wrote:

from OP
****
I've made sure that all the memory
allocated in the [program] is released before making a system() call
to
compress the file and then exiting the script.
The attempt to compress failed because fork returned an -1 error
stating that there is insufficient memory available. It seems C keeps
the memory locked until it ends. Is there away to get C to release
the
memory allocated before it exits?
****

<snip>

> >> Well certainly YES.
>
> > not necessarily in C there isn't.
>
> That would be a surprise to many people since he didn't say
> "automatically".

I fail to see your point. In C there is no guaranteed method to ensure
allocated memory is released for the use of other processes. Hence I
disagreed with the poster who said "certainly YES". So which bit did
I get wrong and what does "automatically" have to do with it?

"BartC" <> writes:

> Marston wrote:
>
>> This file that I'm trying to zip is 504 MB so it needs a lot of
>> memory.
>
> That's surprising, I thought Zip only really needed memory for it's tables,
> and would not be constrained by available memory. Maybe you need a better
> zip program.

It looks like it's bzip2. Like any good compression program, bzip2
compresses the file in blocks, so its memory requirement doesn't grow
with the size of the file. However, by default its baseline memory
requirement is fairly large (about 9MB on my machine). It has an
option -s which reduces it to about 3MB at the expense of compression
ratio.