Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > Using malloc/free in a tight loop

Reply
Thread Tools

Using malloc/free in a tight loop

 
 
Navaneeth
Guest
Posts: n/a
 
      11-17-2010
My application will be looping for 5000 times. On each iteration, it
has to create a string and pass that into other methods for
processing. This string length will only be known at the runtime.

So in each iteration, this is what happens.

ptr = malloc
process(ptr)
free(ptr)

I am wondering doing frequent malloc and free is ok? Will this
fragment the memory?

My development environment is GCC and Linux. But I'd be happy to know
about the implementation in windows also.
 
Reply With Quote
 
 
 
 
Malcolm McLean
Guest
Posts: n/a
 
      11-17-2010
On Nov 17, 11:02*am, Navaneeth <(E-Mail Removed)> wrote:
> My application will be looping for 5000 times. On each iteration, it
> has to create a string and pass that into other methods for
> processing. This string length will only be known at the runtime.
>
> So in each iteration, this is what happens.
>
> ptr = malloc
> process(ptr)
> free(ptr)
>
> I am wondering doing frequent malloc and free is ok? Will this
> fragment the memory?
>
> My development environment is GCC and Linux. But I'd be happy to know
> about the implementation in windows also.
>

Unlikely. When you create strings and delete strings at random you
might fragment the memory, but if you are matching allocations to
frees then the system will usually resuse the memory. (If you print
out the pointers with %p you'll probably find that you get the same
value time after time).

Unless the strings are extremely long then a modern computer isn't
going to notice allocating 5000 of them.

 
Reply With Quote
 
 
 
 
Navaneeth
Guest
Posts: n/a
 
      11-17-2010
> Unless the strings are extremely long then a modern computer isn't
> going to notice allocating 5000 of them.


Thanks. My strings are not long. May be 20 characters.

So in general, can malloc fragment the memory? I was expecting the
modern implementations will be clever enough to reduce this problem.
Isn't that true?
 
Reply With Quote
 
Mikko Rauhala
Guest
Posts: n/a
 
      11-17-2010
On Wed, 17 Nov 2010 04:00:25 -0800 (PST), Navaneeth <(E-Mail Removed)>
wrote:
>> Unless the strings are extremely long then a modern computer isn't
>> going to notice allocating 5000 of them.

>
> Thanks. My strings are not long. May be 20 characters.


You know, if you have a very short maximum cap for the length, you
could just allocate that and reuse the buffer. (You could also by
default reuse the buffer and only reallocate a larger one as it
becomes necessary.)

That said, the first response was correct in that it's not likely to
make much of a difference in this quite simple case. In general though,
I would avoid unnecessary repeated allocations and frees in a tight
loop.

> So in general, can malloc fragment the memory?


In general, it can indeed. Especially if your processing step here
involves allocating space that _isn't_ freed, it can happen in your
example too, but probably not to a large extent. (The malloc
implementation just needs to be smart enough to reuse the space
left by the previous freed buffer, in the common case that it's
sufficiently large).

--
Mikko Rauhala <(E-Mail Removed)> - http://www.iki.fi/mjr/blog/
The Finnish Pirate Party - http://piraattipuolue.fi/
World Transhumanist Association - http://transhumanism.org/
Singularity Institute - http://singinst.org/
 
Reply With Quote
 
JohnF
Guest
Posts: n/a
 
      11-17-2010
Navaneeth <(E-Mail Removed)> wrote:
> My application will be looping for 5000 times. On each iteration, it
> has to create a string and pass that into other methods for
> processing. This string length will only be known at the runtime.
> So in each iteration, this is what happens.
> ptr = malloc
> process(ptr)
> free(ptr)
> I am wondering doing frequent malloc and free is ok? Will this
> fragment the memory?


As per MM's reply, not a problem. Nevertheless, I usually do
this kind of thing, when I very temporarily need lots of short
strings, with a short function that allocates memory from a
static array used as a wraparound buffer, something like
as follows,
#define BUFFSZ 1000 /* any size you like */
void *tempalloc(int nbytes) {
static unsigned char buffer[BUFFSZ]; /* buffer */
static int index = 0; /*start at beginning of buffer*/
void *p = NULL; /* returned ptr */
if ( nbytes <=0 || nbytes > BUFFSZ ) goto end_of_job; /* error */
if ( index+nbytes > BUFFSZ ) index=0; /* wrap buffer */
p = (void *)(buffer+index); /* ptr returned to caller */
index += nbytes; /* next ptr past returned nbytes */
end_of_job: return(p); } /* back to caller */
No need to free, or anything. You just have to be logically sure
you won't still be using the memory when it's "realloc'ed" after
the internal static buffer wraps around. (And the above code
doesn't align returned memory on anything.)
--
John Forkosh ( mailto: http://www.velocityreviews.com/forums/(E-Mail Removed) where j=john and f=forkosh )
 
Reply With Quote
 
Navaneeth
Guest
Posts: n/a
 
      11-17-2010
> * *#define BUFFSZ 1000 */* any size you like */
> * *void *tempalloc(int nbytes) {
> * * *static unsigned char buffer[BUFFSZ]; /* buffer */
> * * *static int index = 0; * * */*start at beginning of buffer*/
> * * *void *p = NULL; * * * * * */* returned ptr */
> * * *if ( nbytes <=0 || nbytes > BUFFSZ ) goto end_of_job; /* error */
> * * *if ( index+nbytes > BUFFSZ ) index=0; /* wrap buffer */
> * * *p = (void *)(buffer+index); /* ptr returned to caller */
> * * *index += nbytes; * * * * * /* next ptr past returned nbytes */
> * * *end_of_job: return(p); } * /* back to caller */


John,

Thanks. This seems to be a good idea. I will evaluate it.
 
Reply With Quote
 
William Hughes
Guest
Posts: n/a
 
      11-17-2010
On Nov 17, 9:01*am, JohnF <(E-Mail Removed)> wrote:
> Navaneeth <(E-Mail Removed)> wrote:
> > My application will be looping for 5000 times. On each iteration, it
> > has to create a string and pass that into other methods for
> > processing. This string length will only be known at the runtime.
> > So in each iteration, this is what happens.
> > * ptr = malloc
> > * process(ptr)
> > * free(ptr)
> > I am wondering doing frequent malloc and free is ok? Will this
> > fragment the memory?

>
> As per MM's reply, not a problem. Nevertheless, I usually do
> this kind of thing, when I very temporarily need lots of short
> strings, with a short function that allocates memory from a
> static array used as a wraparound buffer, something like
> as follows,



A variant that will deal with long
strings (this is the method described
by Mikko Rauhala). The idea is malloc once and reuse
the buffer. Only realloc if you have to.

(untested, no error checking)

* *#define BUFFSZ 20 */* any size you like, 2 will do */
#define MEM_INCREMENT 2 /* unless you have a good reason use 2 */

* *void *tempalloc(int nbytes) {
* * static void *p=malloc(BUFFSZ);
static int currbufsize = BUFFSZ;
* * *

while ( nbytes > currbufsize) {
curbuffsize = currbufsize*MEM_INCREMENT;
p==remlloc(p,currbuufsize);
}

return p;
}


This has the advantage of portable alignment and
it will not fail if a large allocation is requested.
It has the disadvantage that it
requires the use of the library malloc.
You can only use tempalloc safely if you are
sure that you do not need the memory from
a previous call to tempalloc any more.


A
 
Reply With Quote
 
Chris M. Thomasson
Guest
Posts: n/a
 
      11-17-2010
"Navaneeth" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> My application will be looping for 5000 times. On each iteration, it
> has to create a string and pass that into other methods for
> processing. This string length will only be known at the runtime.
>
> So in each iteration, this is what happens.
>
> ptr = malloc
> process(ptr)
> free(ptr)
>
> I am wondering doing frequent malloc and free is ok? Will this
> fragment the memory?
>
> My development environment is GCC and Linux. But I'd be happy to know
> about the implementation in windows also.


Sounds like a region allocator might help you out a bit:

http://groups.google.com/group/comp....a65e8d96f6007c


 
Reply With Quote
 
Chris M. Thomasson
Guest
Posts: n/a
 
      11-18-2010
"Chris M. Thomasson" <(E-Mail Removed)> wrote in message
newseUEo.35278$(E-Mail Removed)...
> "Navaneeth" <(E-Mail Removed)> wrote in message
> news:(E-Mail Removed)...
>> My application will be looping for 5000 times. On each iteration, it
>> has to create a string and pass that into other methods for
>> processing. This string length will only be known at the runtime.
>>
>> So in each iteration, this is what happens.
>>
>> ptr = malloc
>> process(ptr)
>> free(ptr)
>>
>> I am wondering doing frequent malloc and free is ok? Will this
>> fragment the memory?
>>
>> My development environment is GCC and Linux. But I'd be happy to know
>> about the implementation in windows also.

>
> Sounds like a region allocator might help you out a bit:



Or, you might be able to do something really simple like:

http://codepad.org/kTdySzAe
__________________________________________________ ____________
#include <stdio.h>
#include <stdlib.h>




#define ITERS 128U

#define RAND() ((size_t)rand())

#define STRING_MAX_SIZE 64U

#define STRING_ALPHABET "ABCDEFGHIJKLMNOPQRSTUVWXYZ"

#define STRING_ALPHABET_GET() \
STRING_ALPHABET[RAND() % (sizeof(STRING_ALPHABET) - 1)]




char*
string_populate(
char* string,
size_t* psize
){
size_t i;
size_t size = RAND() % STRING_MAX_SIZE;
*psize = size;

string[size] = '\0';

for (i = 0; i < size; ++i)
{
string[i] = STRING_ALPHABET_GET();
}

return string;
}


void
string_process(
char const* string,
size_t size
){
size_t i;

for (i = 0; i < size; ++i)
{
putchar(string[i]);
}

puts("\n__________________________________________ _____\n");
}




int
main(void)
{
char buffer[STRING_MAX_SIZE];
size_t i;

for (i = 0; i < ITERS; ++i)
{
size_t size;
char* string = string_populate(buffer, &size);
string_process(string, size);
}

puts("\n\n________________________________________ ___\n"
"The program has completed.\n");

getchar();

return 0;
}

__________________________________________________ ____________




No need to call `malloc()/free()' and friends... ;^)


 
Reply With Quote
 
Nobody
Guest
Posts: n/a
 
      11-19-2010
On Wed, 17 Nov 2010 04:00:25 -0800, Navaneeth wrote:

> So in general, can malloc fragment the memory? I was expecting the
> modern implementations will be clever enough to reduce this problem.
> Isn't that true?


They will try to minimise it, but it isn't always possible.

E.g.:

for (n = n0; n <= n1; n++) {
void *p = malloc(n);
void *q = malloc(1);
...
free(p);
}

Each new allocation will be 1 byte too large to fit into the space
for the previous allocation. Some of them will fit due to the previous
allocation having been oversized, but they can't all fit.

If you really need to prevent fragmentation, you need a relocating
allocator, which implies a different interface. With malloc(), any
returned pointer remains valid until free()d; the implementation can't
just move blocks as it chooses.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Triple nested loop python (While loop insde of for loop inside ofwhile loop) Isaac Won Python 9 03-04-2013 10:08 AM
performance of tight loop gry Python 8 12-14-2010 11:59 AM
Nice a tight loop? Joseph C Programming 16 07-07-2006 09:11 PM
Update a status object while in a tight loop ? Richard A. DeVenezia Javascript 1 04-04-2004 08:32 AM
70-297 - tight time scales! Marlin (the PFY) Munrow MCSE 0 01-17-2004 11:12 AM



Advertisments