Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > Software memory management reengineering

Reply
Thread Tools

Software memory management reengineering

 
 
Eric Sosman
Guest
Posts: n/a
 
      10-13-2007
Richard Heathfield wrote:
> jacob navia said:
>
> <snip>
>
>> A transient program, as I have explained several times already, leaves
>> its cleanup to the operating system. Thus, it can be called thousands
>> of times, it will always cleanup implicitly.

>
> I have already given an example, upthread, of what you call a "transient"
> program which became a "non-transient" program five years later. Skimp on
> the housekeeping now, and pay for it many times over in the future.


Engineering is about balancing considerations that are
sometimes in conflict. No one consideration or goal is always
paramount. Besides, even when "The program must be easy to
modify" is important, the designer faces the challenge of
trying to anticipate the likely modifications. Sometimes
the crystal ball works, sometimes it's cloudy, and hindsight
is always twenty-twenty.

>> I can repeat this again and again and apparently there will be always
>> the same "argument"...

>
> No matter how many times you repeat an incorrect argument, it remains
> incorrect.


"Incorrect" is too absolute a term to apply here. Can
you think of a good reason for the Unix `shutdown' program
to free() its allocated memory before returning? Yes, a
leaky `shutdown' would be hard to use in a loop, but ...

For what it's worth, many of my own programs follow the
general line Jacob describes, at least in part. "Library"
or "utility" functions are scrupulous about their memory
bookkeeping because they don't know the context they run in;
they're careful about memory for the same reasons they don't
call abort(). But if there's a bunch of long-lived data
that represents the program's "global state," I see nothing
wrong with letting exit() clean it up.

--
Eric Sosman
http://www.velocityreviews.com/forums/(E-Mail Removed)lid
 
Reply With Quote
 
 
 
 
Richard Harter
Guest
Posts: n/a
 
      10-13-2007
On Sun, 14 Oct 2007 07:56:40 +1300, Ian Collins
<(E-Mail Removed)> wrote:

>jacob navia wrote:
>> Richard Heathfield wrote:
>>
>>> The point is that you have to think about this stuff up front.

>>
>> This is wrong.
>>
>> There is absolutely no need to over engineer a program, making it manage
>> memory up front. You allocate memory without freeing it. If this ever
>> becomes a problem you add a pool based strategy or a garbage collector.
>>

>I can see you have never worked on embedded systems.
>
>> Utilities that produce directory listings, linkers, compilers, filters,
>> text searching tools, there are many examples of transient programs.
>> In many of them, ignoring memory allocation problems makes them faster,
>> leaner, and less error prone, since all the bugs associated with
>> "free()" calls can be avoided.
>>

>You fail to realise that your leaky directory listing routine may get
>invoked recursively in something like a backup utility, or get called
>hundreds or even thousands of times in a test suite. Just clean up your
>mess, it isn't that hard.


While what you say is quite true, you are talking about something
different than what Jacob is talking about. Jacob is talking
about transient *programs*, which you changed into *routines*.
If I write a program that lists a single directory specified in
its calling sequence then it can't be "invoked recursively". The
backup utility executes the program multiple times, each
execution being a separate process (or equivalent concept in the
OS of your choice).

Jacob's notion is plausible on the surface - if you have a
collection of utilities that are bounded in their own right and
you connecting them in scripts, then one could save execution
time by not bothering to free storage. That said, there are
several objections that occur to me.

The first is that people do take standalone programs and convert
them to internal functions. What was acceptable in the original
is not in the conversion. Secondly, it is not necessarily the
case that a simple standalone program will have bounded memory
usage. For example, a program that reads lines from stdin and
prints will exhaust memory if a new buffer is allocated for each
line. Thirdly the savings in execution time are strictly
nominal; the costs of starting up and terminating these little
transient programs dominates the cost of freeing storage.



Richard Harter, (E-Mail Removed)
http://home.tiac.net/~cri, http://www.varinoma.com
But the rhetoric of holistic harmony can generate into a kind of
dotty, Prince Charles-style mysticism. -- Richard Dawkins
 
Reply With Quote
 
 
 
 
Richard Heathfield
Guest
Posts: n/a
 
      10-13-2007
Eric Sosman said:

> Richard Heathfield wrote:
>> jacob navia said:
>>
>> <snip>
>>
>>> A transient program, as I have explained several times already, leaves
>>> its cleanup to the operating system. Thus, it can be called thousands
>>> of times, it will always cleanup implicitly.

>>
>> I have already given an example, upthread, of what you call a
>> "transient" program which became a "non-transient" program five years
>> later. Skimp on the housekeeping now, and pay for it many times over in
>> the future.

>
> Engineering is about balancing considerations that are
> sometimes in conflict. No one consideration or goal is always
> paramount.


Certainly true, but I would not classify "can't be bothered to manage
resources properly" as an engineering consideration, any more than I would
classify "can't be bothered to wash the dishes" as being a household
management consideration.

<snip>

>> No matter how many times you repeat an incorrect argument, it remains
>> incorrect.

>
> "Incorrect" is too absolute a term to apply here.


Sorry, Eric, but I can't agree. I consider the argument he is making to be
incorrect.

> Can
> you think of a good reason for the Unix `shutdown' program
> to free() its allocated memory before returning?


Actually, I can't even think of a good reason for 'shutdown' to allocate
memory dynamically.

<snip>

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
 
Reply With Quote
 
Eric Sosman
Guest
Posts: n/a
 
      10-13-2007
Richard Heathfield wrote:
> Eric Sosman said:
>> [...]
>> Can
>> you think of a good reason for the Unix `shutdown' program
>> to free() its allocated memory before returning?

>
> Actually, I can't even think of a good reason for 'shutdown' to allocate
> memory dynamically.


Whether the reason is "good" or not I can't say, but
at least one version certainly does allocate dynamic memory
and does not free it. See line 711 of

http://cvs.opensolaris.org/source/xr...own/shutdown.c

From the copyright notices it appears that this module
is at least twenty-three years old; there are also stylistic
suggestions that its origins antedate the C Standard. Yet in
all that time, it seems no one has felt impelled to embed
`shutdown' as a routine in a larger program that could call
it in a loop! Maybe they would have, were it not for the
memory leak? What an awful thought: a `shutdown' that gobbled
up all the swap space and crashed the system ...

--
Eric Sosman
(E-Mail Removed)lid
 
Reply With Quote
 
Richard Heathfield
Guest
Posts: n/a
 
      10-13-2007
Eric Sosman said:

> Richard Heathfield wrote:
>> Eric Sosman said:
>>> [...]
>>> Can
>>> you think of a good reason for the Unix `shutdown' program
>>> to free() its allocated memory before returning?

>>
>> Actually, I can't even think of a good reason for 'shutdown' to allocate
>> memory dynamically.

>
> Whether the reason is "good" or not I can't say, but
> at least one version certainly does allocate dynamic memory
> and does not free it. See line 711 of
>
>

http://cvs.opensolaris.org/source/xr...own/shutdown.c

<cough>

> From the copyright notices it appears that this module
> is at least twenty-three years old; there are also stylistic
> suggestions that its origins antedate the C Standard. Yet in
> all that time, it seems no one has felt impelled to embed
> `shutdown' as a routine in a larger program that could call
> it in a loop!


I can't imagine why...

> Maybe they would have, were it not for the
> memory leak? What an awful thought: a `shutdown' that gobbled
> up all the swap space and crashed the system ...


Yes, it's very bad practice - if you're unlucky, it might even shut down
the machine. Tut tut.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
 
Reply With Quote
 
Keith Thompson
Guest
Posts: n/a
 
      10-13-2007
Richard Heathfield <(E-Mail Removed)> writes:
[...]
> Certainly true, but I would not classify "can't be bothered to manage
> resources properly" as an engineering consideration, any more than I would
> classify "can't be bothered to wash the dishes" as being a household
> management consideration.

[...]

I don't wash paper plates before I throw them away.

--
Keith Thompson (The_Other_Keith) (E-Mail Removed) <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"
 
Reply With Quote
 
Richard Heathfield
Guest
Posts: n/a
 
      10-14-2007
Keith Thompson said:

> Richard Heathfield <(E-Mail Removed)> writes:
> [...]
>> Certainly true, but I would not classify "can't be bothered to manage
>> resources properly" as an engineering consideration, any more than I
>> would classify "can't be bothered to wash the dishes" as being a
>> household management consideration.

> [...]
>
> I don't wash paper plates before I throw them away.


That's the thing about analogies - they are illustrations, not proofs.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
 
Reply With Quote
 
cr88192
Guest
Posts: n/a
 
      10-14-2007

"jacob navia" <(E-Mail Removed)> wrote in message
news:471022e9$0$5095$(E-Mail Removed)...
> Suppose that you have a module that always allocates
> memory without ever releasing it because the guy that
> wrote it was lazy, as lazy as me.
>
> Now, you want to reuse it in a loop. What do you do?
>
> Contrary to some people that will start crying to that
> &@@""#~ programmer that wrote this sh!!!! you keep
> your cool and you do the following:
>
> 1) Since the program always call malloc and never free()
> you allocate a big chunk of memory and each time that the
> program calls malloc, you replace those calls with another
> that allocates from that memory heap.
>
> 2) When the program finishes you free all memory witha single
> free() call.
>


actually, this is more or less what I do in my "lower compiler" (converts my
RPN-IL or RIL language to assembler).
it is convinient since the memory in question is not shared between runs.
so, I run the basic parts of the compiler, and simply "rewind" the heap when
I am done.

note that my "upper compiler", which works with ASTs, is garbage collected,
and my assembler/linker core, does its own (manual) memory management.


for anyone that knows:
this was also a major memory-management technique in Quake 1 and 2 (maybe Q3
and later, but I have not really investigated this). these engines actually
had 2 memory managers: the hunk (as above), and Z_Malloc (from what I
remember, a circular linked-list, rover, and memory-block based system,
basically, we scan along until we find a free block, and maybe split and
merge as needed, and a rover keeps track of where in the heap we are). the
hunk was used for most game data (maps, models, textures, wordstate, ...),
wheras the Z_Malloc system was used for longer-lived data (such as the
'cvars', or engine variables).

major cost:
one up-front commits a decent-sized chunk of memory to be used in this
manner, and "overflowing" is a potentially fatal condition (in my case, for
example if someone tries to compile some obscenely large source file...).


> Caveats. The program should never call directly
> realloc. Watch for calloc too.
>


the way to deal with realloc, is simply to relocate the object to the end of
the heap...
this can be done much the same as above, a wrapper to the custom allocator
and a memcpy (downsizing though can be made into a no-op).


> I have done this many times. Actually programs that use the
> lazy memory allocation strategy are very easy to convert to a heap
> based strategy if the need ever arises.
>


maybe, maybe not.


> Programs that leak memory but contain a lots of calls to free()
> are much harder to modify like this, since you would have to
> repeat the functionality of free().
>


or, very more the case: code that actually needs to keep track of the memory
after it finishes...
if we need to create, pass around, and return objects, then this approach is
totally ill-suited.


> Of course, a lazy strategy is much easier to upgrade to a GC
> based one. In that case absolutely no code needs to be
> modified. malloc is replaced with GC_malloc or similar,
> and there is nothing else to do.
>


ok.

> --
> jacob navia
> jacob at jacob point remcomp point fr
> logiciels/informatique
> http://www.cs.virginia.edu/~lcc-win32


 
Reply With Quote
 
Malcolm McLean
Guest
Posts: n/a
 
      10-14-2007

"Eric Sosman" <(E-Mail Removed)> wrote in message
>
> But if there's a bunch of long-lived data
> that represents the program's "global state," I see nothing
> wrong with letting exit() clean it up.
>

Writing a clean-up function forces you to keep track of that global data's
pointer structure. If you can't do this easily it is a useful warning that
the program might be getting out of control.

--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

 
Reply With Quote
 
Joachim Schmitz
Guest
Posts: n/a
 
      10-14-2007
"Richard Heathfield" <(E-Mail Removed)> schrieb im Newsbeitrag
news:(E-Mail Removed)...
> Keith Thompson said:
>
>> Richard Heathfield <(E-Mail Removed)> writes:
>> [...]
>>> Certainly true, but I would not classify "can't be bothered to manage
>>> resources properly" as an engineering consideration, any more than I
>>> would classify "can't be bothered to wash the dishes" as being a
>>> household management consideration.

>> [...]
>>
>> I don't wash paper plates before I throw them away.

>
> That's the thing about analogies - they are illustrations, not proofs.

It's not a bad analogy though: if you want to reuse the dish (or code), do
the washing (or free()ing), if you don't, just dump it.

It's quite easy to tell the difference between paper plates and china, it's
not equally easy to tell reusable code from non-reusable though.

Of course I see what you're getting at and I also do have an example were
proper houskeeping would have helped a lot: it was webserver (the NCSA one I
think), which in it's classical form did a fork() on an incoming request,
let the child deal with the reqest and that child just exits when done, not
explicitly free()ing it's memory. Now ao another platform fork() was rather
expensive so the webserver got rewtittion into a long running service,
handling all the request itself (actually a bunch of processes). What
happened then was, that it was leaking memory all over the place and had
largly to be rewritten and redesigned.

Bye, Jojo


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
reengineering c++ code that uses pthreads on Sun Solaris My Python C++ 7 06-03-2011 01:39 AM
Project management / bug management Floris van Haaster ASP .Net 3 09-23-2005 08:36 PM
CatOS web management or CiscoView management ? Martin Bilgrav Cisco 1 12-20-2003 01:49 PM
perl memory management - does @array = () free the memory? Matt Oefinger Perl Misc 0 06-25-2003 09:11 PM



Advertisments