Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C++ > Map erase does not free memory

Reply
Thread Tools

Map erase does not free memory

 
 
Alain Ketterlin
Guest
Posts: n/a
 
      02-07-2013
Ged85 <(E-Mail Removed)> writes:

> On Thursday, February 7, 2013 4:19:04 PM UTC+1, Scott Lurndal wrote:

[...]
>> I'd allocate a pool of anonymous memory with mmap() on a POSIX system
>> and free it with munamp() when no longer needed.


> Most of these structures take just few items, but each time that I
> store a new key/value pair ,deleting some other one, the memory is not
> deallocated. So maps keep growing filling all my 24GB server
> memory....


This should not happen, deallocated blocks should be reused for new
allocations.

> May you be clearer about mmmap()? You're suggesting a custom map
> implementation?


Scott probably thinks about a custom _allocator_ using a block of memory
obtained via mmap(). If you're really alternating allocations and
deallocations it will not help much. It would be useful if you had a big
map that you'd have to deallocate in one shot.

-- Alain.
 
Reply With Quote
 
 
 
 
Alain Ketterlin
Guest
Posts: n/a
 
      02-07-2013
Ged85 <(E-Mail Removed)> writes:

> On Thursday, February 7, 2013 4:28:58 PM UTC+1, Alain Ketterlin wrote:


>> If you're on Linux, see malloc_trim() and friends.


> How can I exploit malloc_trim() function? Custom allocator for map
> classes?


If you use g++'s standard allocator, all variants of new use malloc(),
malloc_trim() should apply directly.

Make sure you read malloc_trim() manual and check whether it applies to
your case. It will help only in specific cases (basically, no allocated
block ends up "above" your map cells and is retained until after
deallocating the map cells).

-- Alain.
 
Reply With Quote
 
 
 
 
Marcel Müller
Guest
Posts: n/a
 
      02-07-2013
On 07.02.13 16.32, Ged85 wrote:
> I'm using ubuntu 10.04 with gcc 4.4.3.
>
> The problem is that I'm writing a network simulator. Simulations are fairly long, and I use maps for implementing its critical data structures.
>
> Most of these structures take just few items, but each time that I store a new key/value pair ,deleting some other one, the memory is not deallocated. So maps keep growing filling all my 24GB server memory....


First of all you should check your application for memory leaks. Maybe
not the map itself leaks memory.

If you are really sure that have no memory leak, then such symptoms are
often caused by heap fragmentation. Allocation and deleting objects of
increasing size can cause the virtual address space to get fragmented.
Especially if the allocations are intercepted by a few long lived objects.
In worst case the memory will grow quadratic with the allocated bytes.
Building large matrices in Matlab row by row was a common example.

> May you be clearer about mmmap()? You're suggesting a custom map implementation?


I do not think that you need to write your own map. It might be
sufficient to use a custom allocator class to separate long lived and
short lived objects in different heaps.


Marcel
 
Reply With Quote
 
Jorgen Grahn
Guest
Posts: n/a
 
      02-07-2013
On Thu, 2013-02-07, Melzzzzz wrote:
> On Thu, 07 Feb 2013 15:19:04 +0000, Scott Lurndal wrote:
>
>>
>> I'd allocate a pool of anonymous memory with mmap() on a POSIX system
>> and free it with munamp() when no longer needed.

>
> That is what malloc does anyway...


On which system? In Linux it's optional, user-configurable, and it's
undocumented whether free() unmaps the memory. It's also only for
huge blocks of data, and unlikely to affect a std::map<K, T>.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
 
Reply With Quote
 
Jorgen Grahn
Guest
Posts: n/a
 
      02-07-2013
On Thu, 2013-02-07, Andy Champ wrote:
> On 07/02/2013 15:58, Melzzzzz wrote:
>> On Thu, 07 Feb 2013 07:32:08 -0800, Ged85 wrote:
>>
>>>
>>> Most of these structures take just few items, but each time that I store
>>> a new key/value pair ,deleting some other one, the memory is not
>>> deallocated. So maps keep growing filling all my 24GB server memory....

>>
>> Are you sure you don't have memory leak somewhere?
>> map should reuse memory , it would grow to some extent but 24gb?
>>

>
> 10 gives you one that's it. The map should grow when you put things in,
> but if you drop things and then put new ones in your memory should stay
> the same. Somewhere you are allocating memory (or possibly creating some
> kind of OS resource?) and not freeing it.
>
> Does your platform have any heap walking diagnostics that would allow
> you to see what is allocated after you have emptied the map?


He said he was on Linux, so chances are he has 'valgrind --memcheck
--leak-check=full'. Should be enough. (No doubt there are other ways
too, but he might as well see his other valgrind-detectable bugs now.)

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
 
Reply With Quote
 
Öö Tiib
Guest
Posts: n/a
 
      02-08-2013
On Thursday, 7 February 2013 17:32:08 UTC+2, Ged85 wrote:
>
> The problem is that I'm writing a network simulator. Simulations are fairly
> long, and I use maps for implementing its critical data structures.


Fork such long separate calculation as a separate process. After it has
simulated just terminate the process. Operating system gets everything
that you did leak back.

> Most of these structures take just few items, but each time that I store a
> new key/value pair ,deleting some other one, the memory is not deallocated.
> So maps keep growing filling all my 24GB server memory....


Freeing dynamically allocated memory does not result with process
returning the memory to operating system. That is normal. Process that
needed memory may need it again. Put your program into cycle of 100 and
see if it takes 100 times more memory.

I am about 98% sure that your program has defects and leaks memory. Try to
nail down your own defects first and then search defects in std::map.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
how does map take care invalid range for erase()? newbie C++ 2 07-27-2007 06:21 PM
Map erase memory problem Milind C++ 1 06-15-2006 11:39 AM
erase vs. erase al.cpwn@gmail.com C++ 7 03-30-2006 11:45 AM
How do i erase all information in my orkut and erase the link to the orkut account? mountbatten@gmail.com Computer Support 1 10-31-2005 12:03 AM
How do I erase a file that wont let me erase it? Dale Custer Computer Support 4 07-06-2003 09:48 AM



Advertisments