Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C++ > Using virtual memory and/or disk to save reduce memory footprint

Reply
Thread Tools

Using virtual memory and/or disk to save reduce memory footprint

 
 
nick
Guest
Posts: n/a
 
      03-05-2009
Hi,

I am writing a C++ GUI tool. I sometimes run out of memory (exceed the
2GB limit) on a 32bit Linux machine. I have optimized my GUI's
database a lot (and still working on it) to reduce the runtime memory
footprint.

I was thinking of ways to off-load part of the database to virtual
memory or disk and reading it back in when required. Does anyone out
there have any papers or suggestions in that direction?

Regards
Nick
 
Reply With Quote
 
 
 
 
David Schwartz
Guest
Posts: n/a
 
      03-05-2009
On Mar 4, 4:59*pm, nick <(E-Mail Removed)> wrote:

> I am writing a C++ GUI tool. I sometimes run out of memory (exceed the
> 2GB limit) *on a 32bit Linux machine. I have optimized my GUI's
> database a lot (and still working on it) to reduce the runtime memory
> footprint.


> I was thinking of ways to off-load part of the database to virtual
> memory or disk and reading it back in when required. Does anyone out
> there have any papers or suggestions in that direction?


You are running out of process virtual address space. You want to find
techniques that conserve process virtual address space. So the first
question is, what's using up all your address space?

If it's memory that you've allocated, you need to allocate less
memory. One solution might be to use a file on disk instead of memory
mapped space. A memory-mapped file consumes process virtual address
space, but if you use 'pread' and 'pwrite' instead, no vm space is
needed.

DS
 
Reply With Quote
 
 
 
 
nick
Guest
Posts: n/a
 
      03-05-2009
Thanks for your tips David! My tool is schematic driven PCB design
tool. There are multiple levels of schematics that can be composed.
In my profiling, most of the memory is taken up during the elaboration
of the various schematics i.e. storing of the various design-related
informations and then connecting them up.


Regards
Nick
 
Reply With Quote
 
David Schwartz
Guest
Posts: n/a
 
      03-05-2009
On Mar 4, 5:49*pm, nick <(E-Mail Removed)> wrote:

> Thanks for your tips David! My tool is schematic driven PCB design
> tool. There are multiple levels of schematics that can be composed.
> In my profiling, most of the memory is taken up during the elaboration
> of the various schematics i.e. storing of the various design-related
> informations and then connecting them up.


Rather than storing them in memory, why not store them in a file on
disk?

If there's enough physical memory, the file will stay in cache anyway.
If there isn't enough physical memory, trying to keep it in memory
would result in it swapping to disk anyway.

So it should be roughly performance neutral, but save a lot of vm
space.

DS
 
Reply With Quote
 
nick
Guest
Posts: n/a
 
      03-05-2009
> Why are you flattening the whole thing in memory? *I'm not criticizing,
> just asking. *(I spent about 6 months working full-time on a schematic
> editor for VLSI designs.)


Yes, you are right. Do you have any publications that I can look up or
suggestions on how to avoid elaborating everything in memory?

I know what I have just asked would probably be proprietary
information but if there is any public domain information that you
could point me to, I would be eternally grateful

Regards
Nick
 
Reply With Quote
 
tharinda.gl@gmail.com
Guest
Posts: n/a
 
      03-05-2009


nick wrote:
> > Why are you flattening the whole thing in memory? *I'm not criticizing,
> > just asking. *(I spent about 6 months working full-time on a schematic
> > editor for VLSI designs.)

>
> Yes, you are right. Do you have any publications that I can look up or
> suggestions on how to avoid elaborating everything in memory?
>
> I know what I have just asked would probably be proprietary
> information but if there is any public domain information that you
> could point me to, I would be eternally grateful
>
> Regards
> Nick


I remember somebody suggest me this site, this might be helpful to
your work

http://stxxl.sourceforge.net/

Haven't used it though
 
Reply With Quote
 
James Kanze
Guest
Posts: n/a
 
      03-05-2009
On Mar 5, 3:20 am, Jeff Schwab <(E-Mail Removed)> wrote:
> nick wrote:
> > I am writing a C++ GUI tool. I sometimes run out of memory
> > (exceed the 2GB limit) on a 32bit Linux machine.


> What 2GB limit? Do you mean that the machine only has 2GB of
> RAM? Or do you mean that you're exceeding the 3 or 4 GB limit
> on address space?


I was wondering about that myself. I've had programs on a 32
bit Linux that used well over 3 MB.

> > I have optimized my GUI's database a lot (and still working
> > on it) to reduce the runtime memory footprint.


> > I was thinking of ways to off-load part of the database to
> > virtual memory or disk and reading it back in when required.


> The kernel already does that for you, automatically. Doing it manually
> is called "overlaying."


> http://en.wikipedia.org/wiki/Overlay_(programming)


The kernel can only do it when the entire image would fit into
the virtual address space (4 MB under 32 bit Linux).
"Overlaying" will allow a lot more. And it's only called
overlaying when you swap in and out code and named variables; if
you're just buffering data, the name doesn't apply. (As an
extreme example, programs like grep or sed easily work on data
sets that are in the Gigabyte range or larger; they only hold a
single line in memory at a time, however. And I'm sure you
wouldn't call this overlaying.)

FWIW: I don't think that the Linux linker supports overlay
generation, at least in the sense I knew it 25 or 30 years ago.
(Although explicitly load and unload dynamic objects, that
probably comes out to the same thing.)

--
James Kanze (GABI Software) email:(E-Mail Removed)
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
 
Reply With Quote
 
Benjamin Rampe
Guest
Posts: n/a
 
      03-06-2009
James Kanze schrieb:
>> [..] Or do you mean that you're exceeding the 3 or 4 GB limit
>> on address space?


> I was wondering about that myself. I've had programs on a 32
> bit Linux that used well over 3 MB.

(Probably 3GB...) PAE enabled?
http://en.wikipedia.org/wiki/Physical_Address_Extension

Regards,
Benjamin
 
Reply With Quote
 
James Kanze
Guest
Posts: n/a
 
      03-06-2009
On Mar 5, 3:05 pm, Jeff Schwab <(E-Mail Removed)> wrote:
> James Kanze wrote:
> > FWIW: I don't think that the Linux linker supports overlay
> > generation, at least in the sense I knew it 25 or 30 years ago.


[...]
> Oddly, those kinds of overlays (IIUC) are coming back into
> fashion, for processors with lots of little cores, each with
> its own small, dedicated cache. On cell, the loading and
> unloading have to be managed manually, by the programmer.


I don't know what the current situation is. (My current
applications run on dedicated machines with enough real memory
that they never have to page out.) But I remember talking to
people who worked on the mainframe Fortran when virtual memory
was being introduced; their experience was that programs using
virtual memory, instead of overlays, were often several orders
of magnitude slower. When you had to explicitly load an
overlay, you thought about it, and did it as infrequently as
possible. When the system decides to load or unload a virtual
page is invisible to you, and you can end up with a lot more
paging than you wanted. (On the other hand, increase the size
of the real memory, and the virtual memory based system will
cease paging entirely. Where as the overlays will still by
loaded and unloaded each time you request it.)

--
James Kanze (GABI Software) email:(E-Mail Removed)
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
 
Reply With Quote
 
James Kanze
Guest
Posts: n/a
 
      03-06-2009
On Mar 6, 3:02 am, Benjamin Rampe <(E-Mail Removed)> wrote:
> James Kanze schrieb:
>
> >> [..] Or do you mean that you're exceeding the 3 or 4 GB limit
> >> on address space?

> > I was wondering about that myself. I've had programs on a 32
> > bit Linux that used well over 3 MB.


> (Probably 3GB...)


Yes.

> PAE enabled?
> http://en.wikipedia.org/wiki/Physical_Address_Extension


No. 3GB easily fits in the address space of a 32 bit processor.

--
James Kanze (GABI Software) email:(E-Mail Removed)
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Performance and footprint of virtual function George2 C++ 1 01-31-2008 09:17 AM
JMS memory footprint size Beatrice Rutger Java 0 06-05-2005 09:56 PM
better to compress the jpeg or reduce resolution or reduce pixel size? Mr.Will Digital Photography 8 10-08-2004 03:16 PM
look for C++ "Keith Gorlen smalltalk like classes" or NIHCL to reduce memory footprint used by STL francois C++ 1 12-05-2003 02:27 PM
Huge memory footprint using SOAP extensions! R. Sterrenburg ASP .Net Web Services 0 12-02-2003 03:31 PM



Advertisments