Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > Competing criteria

Reply
Thread Tools

Competing criteria

 
 
nroberts
Guest
Posts: n/a
 
      10-21-2011
I'm having trouble figuring out how to get seemingly mutually
exclusive goals and criteria to work together.

The technical lead/architect for the project I'm on is from an
embedded background and therefore doesn't like heap allocation. I
don't know if this is a real issue in that domain, having never done
it myself, but that's what he's told me when complaining about my use
of allocated buffers and such. Factory like functions also seem to
upset him. I'm not so sure about function pointers, but every time
I've made something that wasn't of the direct call, stack or global
variable kind of design he's kind of freaked out on me.

At the same time he wants me to write reusable code and has previously
wanted me to make unit tests, though now I'm not even sure about that
since he won't even look at them.

I'm getting kind of ****ed off to tell the truth, but I still want to
try.

I'm coming into a project where the best solution seems to be a
multithreaded service. It's basically a proxy for a soap server
created by a third party that we're making generic and less soapy in
order for our various products to use it. Some of the calls can take
a long time and writing it multithreaded allows us to use the gsoap
library without having to muck about to use it asynchronously as it
doesn't really support such use.

I created a prototype for this service in C++ using all the
allocation, RAII, polymorphism, etc...so I could get a clear picture
of the problem. Now I need to convert the design into C (I'm fairly
certain he's going to tell me to). Normally this would be fairly
easy, replace exception processing with error codes, rewrite std::list
and such for all the types I want to hold or to use void*...use opaque
types and createXXX() functions... I'm not doing anything
particularly incredible with the C++, it could almost as easily be C.
This kind of stuff doesn't fly here though.

I can create circular buffer stacks to handle things like my thread
pools instead of allocating stack nodes. I can just declare my opaque
types openly and have initXXX() functions. Etc...I think this kind of
thing would make the guy happy but I don't see how to do that while
also meeting another goal he's said I need to meet: make this thing so
that it's a generic architecture for any soap connecting proxy. I
can't think of how to do such circular buffer stacks for specific
types (because I can't allocate) and still write them in a way that
can be used in different ways.

How does one make reusable components without at least some generics
and heap allocations? When everything has to be a direct call, how do
you allow clients to override behavior? What are some idioms,
patterns, whatever that I can use to at least make some attempt at
getting both of these ideals to work together?
 
Reply With Quote
 
 
 
 
Seebs
Guest
Posts: n/a
 
      10-21-2011
On 2011-10-21, nroberts <(E-Mail Removed)> wrote:
> I'm having trouble figuring out how to get seemingly mutually
> exclusive goals and criteria to work together.


Heh.

> The technical lead/architect for the project I'm on is from an
> embedded background and therefore doesn't like heap allocation.


This is probably an obsolete view, but who knows? Anyway, the canonical
solution to "not heap allocation" is to define a reasonable maximum count
of simultaneous objects, statically allocate them, and then basically
write your own mini-allocator that keeps track of which ones you're using.

This, you will note, is probably stupid.

Note that either way, it is very important that you handle lack of
resources gracefully.

-s
--
Copyright 2011, all wrongs reversed. Peter Seebach / http://www.velocityreviews.com/forums/(E-Mail Removed)
http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures
http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
I am not speaking for my employer, although they do rent some of my opinions.
 
Reply With Quote
 
 
 
 
John Gordon
Guest
Posts: n/a
 
      10-21-2011
In <(E-Mail Removed)> nroberts <(E-Mail Removed)> writes:

> At the same time he wants me to write reusable code and has previously
> wanted me to make unit tests, though now I'm not even sure about that
> since he won't even look at them.


He wants to ensure that adequate tests exist but doesn't care about the
gory details; that's not necessarily an unreasonable attitude for a project
lead.

--
John Gordon A is for Amy, who fell down the stairs
(E-Mail Removed) B is for Basil, assaulted by bears
-- Edward Gorey, "The Gashlycrumb Tinies"

 
Reply With Quote
 
Stefan Ram
Guest
Posts: n/a
 
      10-21-2011
nroberts <(E-Mail Removed)> writes:
>At the same time he wants me to


If he is contradicting himself, you cannot succeed. So to
ensure you are not blamed for his contradictions yourself,
you might consider asking him for his coding requirements
/in writing/.

>I created a prototype for this service in C++ using all the
>allocation, RAII, polymorphism, etc...so I could get a clear picture


Thinking in C++ and then translating to C will not always
give the best results.

>How does one make reusable components without at least some
>generics and heap allocations?


You can write generic code with preprocessor macros, but
will loose type safety in comparision to C++.

Sometimes, storage with dynamic storage duration can be
replaced by storage with automatic or static storage
duration (sometimes in an outer scope). This depends on the
details. One also can allocate a large chunk of storage of
static or automatic storage duration and then use one's
own custom heap implementation on top of this.

For example, study how TeX was implemented in Pascal.

TeX does all of its dynamic allocation itself from
fixed-size arrays and uses only fixed-point arithmetic
for its internal calculations.

http://en.wikipedia.org/wiki/TeX

>When everything has to be a direct call, how do you allow
>clients to override behavior?


Sometimes this can be replaced by switch statements,
but one will loose the advantages of an OOP-like style.

>What are some idioms, patterns, whatever that I can use to at
>least make some attempt at getting both of these ideals to
>work together?


The best designs are based on the target language right from
the start. For example, when you design a typesetting system
for Java, no one will have fun then translating this to Pascal.
When a programmer with experience in Pascal, designs the
implementation for Pascal right from the start, it will be better.

 
Reply With Quote
 
Richard Damon
Guest
Posts: n/a
 
      10-22-2011
On 10/21/11 5:49 PM, nroberts wrote:
> I'm having trouble figuring out how to get seemingly mutually
> exclusive goals and criteria to work together.
>
> The technical lead/architect for the project I'm on is from an
> embedded background and therefore doesn't like heap allocation. I
> don't know if this is a real issue in that domain, having never done
> it myself, but that's what he's told me when complaining about my use
> of allocated buffers and such. Factory like functions also seem to
> upset him. I'm not so sure about function pointers, but every time
> I've made something that wasn't of the direct call, stack or global
> variable kind of design he's kind of freaked out on me.
>
> At the same time he wants me to write reusable code and has previously
> wanted me to make unit tests, though now I'm not even sure about that
> since he won't even look at them.
>
> I'm getting kind of ****ed off to tell the truth, but I still want to
> try.
>
> I'm coming into a project where the best solution seems to be a
> multithreaded service. It's basically a proxy for a soap server
> created by a third party that we're making generic and less soapy in
> order for our various products to use it. Some of the calls can take
> a long time and writing it multithreaded allows us to use the gsoap
> library without having to muck about to use it asynchronously as it
> doesn't really support such use.
>
> I created a prototype for this service in C++ using all the
> allocation, RAII, polymorphism, etc...so I could get a clear picture
> of the problem. Now I need to convert the design into C (I'm fairly
> certain he's going to tell me to). Normally this would be fairly
> easy, replace exception processing with error codes, rewrite std::list
> and such for all the types I want to hold or to use void*...use opaque
> types and createXXX() functions... I'm not doing anything
> particularly incredible with the C++, it could almost as easily be C.
> This kind of stuff doesn't fly here though.
>
> I can create circular buffer stacks to handle things like my thread
> pools instead of allocating stack nodes. I can just declare my opaque
> types openly and have initXXX() functions. Etc...I think this kind of
> thing would make the guy happy but I don't see how to do that while
> also meeting another goal he's said I need to meet: make this thing so
> that it's a generic architecture for any soap connecting proxy. I
> can't think of how to do such circular buffer stacks for specific
> types (because I can't allocate) and still write them in a way that
> can be used in different ways.
>
> How does one make reusable components without at least some generics
> and heap allocations? When everything has to be a direct call, how do
> you allow clients to override behavior? What are some idioms,
> patterns, whatever that I can use to at least make some attempt at
> getting both of these ideals to work together?


As someone who has written a number of embedded projects, I can give
some grounds for the desire to limit heap usage and where it can
reasonably be used and where it should not.

I typical embedded project can NOT make the common "hosted" assumption
of nearly unlimited memory being available. It also typically needs to
have a virtually infinite run time, so heap fragmentation needs to be a
concern.

The guidelines I tend to work with are:

1) Heap allocations at "start-up" are normally not an issue. If you
don't have the memory needed, you find out right away, not at some
unknown time in the future.

2) Critical tasks, that must be able to complete on a schedule may NOT
make further heap allocation, but only use statically allocated buffers
or buffer allocated at start-up.

2a) If a task only becomes "critical" after some other action that is
allowed to fail, that other action is allowed to allocate the resources
for the critical task, and must fail if it can't get them.

3) Non-critical tasks may be allowed to allocate dynamic resources, but
there should be points in the execution of the program where most of
these have been returned to the heap to clear up heap fragmentation.

With this, your program can be generic (at compile time), as hopefully
you can know which types are needed for this program, and you can then
preallocate the needed buffer set, but the code is still usable for
other applications that use different sets of types.

If you point out that you are only doing heap allocations at "startup"
time, (or for non-critical code tasks, that are allowed to fail/be
delayed), then he will hopefully be satisfied.
 
Reply With Quote
 
Roberto Waltman
Guest
Posts: n/a
 
      10-22-2011
Seebs wrote:
> the canonical
>solution to "not heap allocation" is to define a reasonable maximum count
>of simultaneous objects, statically allocate them, and then basically
>write your own mini-allocator that keeps track of which ones you're using.
>This, you will note, is probably stupid.


Far from being stupid, that is the only way you can guarantee that
memory allocation will not fail due to memory fragmentation, in
systems with limited resources, no virtual memory and that must run
24/7.
--
Roberto Waltman

[ Please reply to the group,
return address is invalid ]
 
Reply With Quote
 
Richard Damon
Guest
Posts: n/a
 
      10-22-2011
On 10/22/11 2:06 PM, Roberto Waltman wrote:
> Seebs wrote:
>> the canonical
>> solution to "not heap allocation" is to define a reasonable maximum count
>> of simultaneous objects, statically allocate them, and then basically
>> write your own mini-allocator that keeps track of which ones you're using.
>> This, you will note, is probably stupid.

>
> Far from being stupid, that is the only way you can guarantee that
> memory allocation will not fail due to memory fragmentation, in
> systems with limited resources, no virtual memory and that must run
> 24/7.
> --
> Roberto Waltman
>


As I mentioned last night, that isn't totally true. It can be generally
considered safe to do heap allocation during initial startup (for
buffers that will then live forever, perhaps handed out and collected
with allocation functions.

While in principle, it would normally be possible to rewrite all the
code that uses the heap to use statically allocated buffers, this often
forces details of implementation to leak out, especially of objects that
need internal buffers that can't be fixed in size over all uses.

It can also be acceptable to heap allocations for non-critical
operations that are allowed to fail or be postponed for arbitrary
periods of time. These allocations should be for a limited period of
time, and it is desirable for there to be occasional times when most of
these are all released at the same time to let the heap defragment.
 
Reply With Quote
 
Phil Carmody
Guest
Posts: n/a
 
      10-22-2011
Roberto Waltman <(E-Mail Removed)> writes:
> Seebs wrote:
> > the canonical
> >solution to "not heap allocation" is to define a reasonable maximum count
> >of simultaneous objects, statically allocate them, and then basically
> >write your own mini-allocator that keeps track of which ones you're using.
> >This, you will note, is probably stupid.

>
> Far from being stupid, that is the only way you can guarantee that
> memory allocation will not fail due to memory fragmentation, in
> systems with limited resources, no virtual memory and that must run
> 24/7.


False, and likely to fail in most embedded scenarios I've worked on.
Likely to fail, as if you have 100 units of RAM, and may need up to 20
of size 5, or 5 of size 20, but not simultaniously, then you must use
dynamic allocation.

And false, because if you never hand out pointers, but handles, then
you can use dynamic memory allocation and compact your heap either in
the background or on demand.

Phil
--
Unix is simple. It just takes a genius to understand its simplicity
-- Dennis Ritchie (1941-2011), Unix Co-Creator
 
Reply With Quote
 
Ark
Guest
Posts: n/a
 
      10-22-2011


On 10/22/2011 4:47 PM, Richard Damon wrote:
> On 10/22/11 2:06 PM, Roberto Waltman wrote:
>> Seebs wrote:
>>> the canonical
>>> solution to "not heap allocation" is to define a reasonable maximum
>>> count
>>> of simultaneous objects, statically allocate them, and then basically
>>> write your own mini-allocator that keeps track of which ones you're
>>> using.
>>> This, you will note, is probably stupid.

>>
>> Far from being stupid, that is the only way you can guarantee that
>> memory allocation will not fail due to memory fragmentation, in
>> systems with limited resources, no virtual memory and that must run
>> 24/7.
>> --
>> Roberto Waltman
>>

>
> As I mentioned last night, that isn't totally true. It can be generally
> considered safe to do heap allocation during initial startup (for
> buffers that will then live forever, perhaps handed out and collected
> with allocation functions.
>
> While in principle, it would normally be possible to rewrite all the
> code that uses the heap to use statically allocated buffers, this often
> forces details of implementation to leak out, especially of objects that
> need internal buffers that can't be fixed in size over all uses.
>
> It can also be acceptable to heap allocations for non-critical
> operations that are allowed to fail or be postponed for arbitrary
> periods of time. These allocations should be for a limited period of
> time, and it is desirable for there to be occasional times when most of
> these are all released at the same time to let the heap defragment.


IOW, malloc() can be (made) more or less benign but free() is totally
unacceptable.
Which is not totally true either.
If memory serves me right, there are allocation algorithms that
guarantee to succeed if the contiguous memory pool is "only" twice as
large as the max you need at any given time. Of course if you already
know what this max is, you can probably do better w.r.t. memory resources.
OTOH, for any allocation algorithm there is a pattern of
allocations/deallocations which would fatally fragment the pool (aka
heap), in the sense that enough total memory is available but it is not
contiguous
- Ark
 
Reply With Quote
 
Richard Damon
Guest
Posts: n/a
 
      10-22-2011
On 10/22/11 5:33 PM, Ark wrote:
>
>
> On 10/22/2011 4:47 PM, Richard Damon wrote:


>>
>> As I mentioned last night, that isn't totally true. It can be generally
>> considered safe to do heap allocation during initial startup (for
>> buffers that will then live forever, perhaps handed out and collected
>> with allocation functions.
>>
>> While in principle, it would normally be possible to rewrite all the
>> code that uses the heap to use statically allocated buffers, this often
>> forces details of implementation to leak out, especially of objects that
>> need internal buffers that can't be fixed in size over all uses.
>>
>> It can also be acceptable to heap allocations for non-critical
>> operations that are allowed to fail or be postponed for arbitrary
>> periods of time. These allocations should be for a limited period of
>> time, and it is desirable for there to be occasional times when most of
>> these are all released at the same time to let the heap defragment.

>
> IOW, malloc() can be (made) more or less benign but free() is totally
> unacceptable.
> Which is not totally true either.
> If memory serves me right, there are allocation algorithms that
> guarantee to succeed if the contiguous memory pool is "only" twice as
> large as the max you need at any given time. Of course if you already
> know what this max is, you can probably do better w.r.t. memory resources.
> OTOH, for any allocation algorithm there is a pattern of
> allocations/deallocations which would fatally fragment the pool (aka
> heap), in the sense that enough total memory is available but it is not
> contiguous
> - Ark


The rule I always use is a task is not allowed to fail, then it can't
call on anything that might fail. malloc, in general, has the
possibility to fail so needs to be avoided in tasks that can not fail.

free() is actually always safe, as free can never fail, and can never
cause another malloc to fail. What can get you in trouble is to free a
resource that you will need in the future, as after you free a memory
block, you can't be sure you can get it back until every block allocated
after it is freed too (based on normal heap rules).

While, as you say, there are patterns of allocations and deallocations
that fragment the heap, the rule that allocations (after the startup
code) are be of short/finite duration is designed it prevent those bad
patterns. Also, the goal to have periods of time when you free most of
the short term usage also minimizes how bad fragmentation gets.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
competing applications and subapplications BriC ASP .Net 0 12-22-2006 05:05 AM
Microsoft Has Stopped Competing with Linux Au79 Computer Support 2 03-23-2006 02:35 PM
Overview of competing Python modules? Thorsten Kampe Python 2 09-16-2004 09:20 PM
SVG: competing dynamic graphic generating modalities BMM XML 0 05-05-2004 09:34 PM
CSS background competing selectors question Derek Clarkson HTML 3 11-07-2003 12:08 AM



Advertisments