Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Python 2.6 still not giving memory back to the OS...

Reply
Thread Tools

Re: Python 2.6 still not giving memory back to the OS...

 
 
Mark Dickinson
Guest
Posts: n/a
 
      08-15-2009
On Aug 15, 12:55*pm, Chris Withers <(E-Mail Removed)> wrote:
> Hi All,
>
> I thought this was fixed back in Python 2.5, but I guess not?
>
> So, I'm playing in an interactive session:
>
> *>>> from xlrd import open_workbook
> *>>> b = open_workbook('some.xls',pickleable=0,formatting_i nfo=1)
>
> At this point, top shows the process usage for python to be about 500Mb.
> That's okay, I'd expect that, b is big
>
> *>>> del b
>
> However, it still does now, maybe the garbage collector needs a kick?
>
> *>>> import gc
> *>>> gc.collect()
> 702614
>
> Nope, still 500Mb. What gives? How can I make Python give the memory its
> no longer using back to the OS?
> [...]


Can you get the same effects without using the xlrd module? I don't
have xlrd installed on my system (OS X 10.5/Intel), but I just tried
the following:

Python 2.6.2 (r262:71600, Jun 17 2009, 09:08:27)
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> b = 'x'*(10**9)
>>> f = open('testfile.txt', 'w')
>>> f.write(b)
>>> del b
>>> f = open('testfile.txt')
>>> b = f.read()
>>> del b


and got the expected memory usage for my Python process, as
displayed by top: memory usage went up to nearly 1Gb after
each assignment to b, then dropped down to 19 Mb or so after
each 'del b'. I get similar results under Python 2.5.

So maybe there's something in xlrd that's hanging on to all
that memory?

Mark
 
Reply With Quote
 
 
 
 
John Machin
Guest
Posts: n/a
 
      08-16-2009
On Aug 16, 2:41*am, Mark Dickinson <(E-Mail Removed)> wrote:
> and got the expected memory usage for my Python process, as
> displayed by top: *memory usage went up to nearly 1Gb after
> each assignment to b, then dropped down to 19 Mb or so after
> each 'del b'. *I get similar results under Python 2.5.
>
> So maybe there's something in xlrd that's hanging on to all
> that memory?


News to me, and news to guppy/heapy v0.1.9 -- unless the new Windows
support has some deficiencies -- after `del b` heapy can't find any
trace of the Book object and its contents.

As far as releasing memory back to the OS is concerned, I have dim
memories of *x systems where free() would return space to the OS if
the block was "large" and it was next to the "break" point ... this
effect could be what you are seeing.
 
Reply With Quote
 
 
 
 
Martin v. L÷wis
Guest
Posts: n/a
 
      08-16-2009
> As far as releasing memory back to the OS is concerned, I have dim
> memories of *x systems where free() would return space to the OS if
> the block was "large" and it was next to the "break" point ... this
> effect could be what you are seeing.


Today, there are two cases when malloc returns memory on a typical
Unix system (in particular, in Linux malloc):
a) if the malloc block block is small (below page size), it is allocated
from the brk heap, where it can only be returned if the last page of
that heap is completely free, and
b) if the block is large (multiple pages), it gets returned to the
system right away.

Case b) can only happen if the C malloc uses mmap to allocate large
blocks. For Python, case b) is realistic, in the sense that most
allocations go to Python's obmalloc, and those allocate from C malloc
in chunks of 256kiB (IIRC). So when such an arena is completely free
(not a single Python object allocated on it anymore), it can get
returned to the system.

In some case, Python also allocates smaller blocks from C malloc; in
this case, a) will trigger. So as long as something at the end of the
heap stays allocated, C malloc will not return anything from the brk
heap to the system.

Regards,
Martin
 
Reply With Quote
 
Chris Withers
Guest
Posts: n/a
 
      08-24-2009
Martin v. L÷wis wrote:
> Today, there are two cases when malloc returns memory on a typical
> Unix system (in particular, in Linux malloc):
> a) if the malloc block block is small (below page size), it is allocated
> from the brk heap, where it can only be returned if the last page of
> that heap is completely free, and
> b) if the block is large (multiple pages), it gets returned to the
> system right away.
>
> Case b) can only happen if the C malloc uses mmap to allocate large
> blocks.


I believe (and John will correct me if I'm wrong) that xlrd uses mmap,
so why am I not seeing the memory being returned?

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
 
Reply With Quote
 
John Machin
Guest
Posts: n/a
 
      08-25-2009
On Aug 25, 2:08*am, Chris Withers <(E-Mail Removed)> wrote:
> Martin v. L÷wis wrote:
> > Today, there are two cases when malloc returns memory on a typical
> > Unix system (in particular, in Linux malloc):
> > a) if the malloc block block is small (below page size), it is allocated
> > * *from the brk heap, where it can only be returned if the last page of
> > * *that heap is completely free, and
> > b) if the block is large (multiple pages), it gets returned to the
> > * *system right away.

>
> > Case b) can only happen if the C malloc uses mmap to allocate large
> > blocks.

>
> I believe (and John will correct me if I'm wrong) that xlrd uses mmap,
> so why am I not seeing the memory being returned?


xlrd uses mmap to access the input file; this seems to have zero
correlation with "the C malloc uses mmap to allocate large blocks".
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: Python 2.6 still not giving memory back to the OS... Dave Angel Python 0 08-15-2009 01:44 PM
GC.Collect() not cleaning memory, how to find out what references to lots of memory still exist? DR ASP .Net 0 04-15-2008 09:03 PM
The giving that keeps on giving sixteenmillion C Programming 0 11-19-2007 10:59 PM
Python not giving free memory back to the os get's me in real problems ... leuchte@gmail.com Python 17 04-26-2007 01:41 PM
Replacement Digital Rebel Still Giving Me Fluctuating Setting Problems digitalrube@yahoo.com Digital Photography 18 12-01-2006 06:15 PM



Advertisments