Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: python memory analysis

Reply
Thread Tools

Re: python memory analysis

 
 
Alexander Hoffmann
Guest
Posts: n/a
 
      09-06-2004
On Monday 06 September 2004 17:05, Miki Tebeka wrote:
Hi Miki,
Thanks for your reply !

> Hello Alexander,
>
> > I wrote a deamon like program in python and after a while I realized that
> > it consumes more and more memory over time. So my question is: How to
> > debug memory leaks in python ?

>
> Try the gc module.


You probably mean to utilize gc.get_objects and build some kind of statics
upon it, don't you ?
Ok, so this means there is indeed no tool which will help me with that ?

-alex
 
Reply With Quote
 
 
 
 
Jeremy Bowers
Guest
Posts: n/a
 
      09-06-2004
On Mon, 06 Sep 2004 16:51:09 +0200, Alexander Hoffmann wrote:
> You probably mean to utilize gc.get_objects and build some kind of statics
> upon it, don't you ?
> Ok, so this means there is indeed no tool which will help me with that ?
>
> -alex


So far, every time I've had a memory leak in Python, simply running the
process until it leaks for a while, then looking at gc.get_objects(), has
made it immediately clear what was being leaked. (Regretfully, I have
never been able to isolate a test case for any of these leaks.) You
usually see a bajillion instances of some class and that has always been
enough of a clue.

Due to how Python works, I'm not sure it is even possible to create a tool
that does any better than that. It is meaningful to talk about a C
function that "leaks memory" by allocating and never deallocating, but it
is much trickier to assign "blame" in Python in a way that could ever have
meaning to a tool. If the object is left lying around it is already
because the computer believes somebody has a reference to it; how is the
computer to "know" this reference isn't "real"? (Assuming no bugs in
Python of course, but even so you're left with the same problem.)
 
Reply With Quote
 
 
 
 
Alexander Hoffmann
Guest
Posts: n/a
 
      09-06-2004
Hello Jeremy,
Thanks for your help !

Indeed a first invocation of gc.get_objects already gave me an idea about what
might go wrong. It seems like the garbage collector does not remove objects a
and b if a references b even though no one else references either a or b.

I will continue my analysis.

Thanks,
Alex

On Monday 06 September 2004 17:38, Jeremy Bowers wrote:
> On Mon, 06 Sep 2004 16:51:09 +0200, Alexander Hoffmann wrote:
> > You probably mean to utilize gc.get_objects and build some kind of
> > statics upon it, don't you ?
> > Ok, so this means there is indeed no tool which will help me with that ?
> >
> >
> > -alex

>
> So far, every time I've had a memory leak in Python, simply running the
> process until it leaks for a while, then looking at gc.get_objects(), has
> made it immediately clear what was being leaked. (Regretfully, I have
> never been able to isolate a test case for any of these leaks.) You
> usually see a bajillion instances of some class and that has always been
> enough of a clue.
>
> Due to how Python works, I'm not sure it is even possible to create a tool
> that does any better than that. It is meaningful to talk about a C
> function that "leaks memory" by allocating and never deallocating, but it
> is much trickier to assign "blame" in Python in a way that could ever have
> meaning to a tool. If the object is left lying around it is already
> because the computer believes somebody has a reference to it; how is the
> computer to "know" this reference isn't "real"? (Assuming no bugs in
> Python of course, but even so you're left with the same problem.)


 
Reply With Quote
 
Chris Liechti
Guest
Posts: n/a
 
      09-06-2004
Alexander Hoffmann <(E-Mail Removed)> wrote in
news:(E-Mail Removed):

> Hello Jeremy,
> Thanks for your help !
>
> Indeed a first invocation of gc.get_objects already gave me an idea
> about what might go wrong. It seems like the garbage collector does
> not remove objects a and b if a references b even though no one else
> references either a or b.


the GC cant remove cycles with objects that contain a __del__ method.
non cycles are removed by reference counting anayway.

either dont make circles or don't use __del__, maybe the weakref module can
be of some use too.

chris


> I will continue my analysis.
>
> Thanks,
> Alex
>
> On Monday 06 September 2004 17:38, Jeremy Bowers wrote:
>> On Mon, 06 Sep 2004 16:51:09 +0200, Alexander Hoffmann wrote:
>> > You probably mean to utilize gc.get_objects and build some kind of
>> > statics upon it, don't you ?
>> > Ok, so this means there is indeed no tool which will help me with
>> > that ?
>> >
>> >
>> > -alex

>>
>> So far, every time I've had a memory leak in Python, simply running
>> the process until it leaks for a while, then looking at
>> gc.get_objects(), has made it immediately clear what was being
>> leaked. (Regretfully, I have never been able to isolate a test case
>> for any of these leaks.) You usually see a bajillion instances of
>> some class and that has always been enough of a clue.
>>
>> Due to how Python works, I'm not sure it is even possible to create a
>> tool that does any better than that. It is meaningful to talk about a
>> C function that "leaks memory" by allocating and never deallocating,
>> but it is much trickier to assign "blame" in Python in a way that
>> could ever have meaning to a tool. If the object is left lying around
>> it is already because the computer believes somebody has a reference
>> to it; how is the computer to "know" this reference isn't "real"?
>> (Assuming no bugs in Python of course, but even so you're left with
>> the same problem.)

>
>




--
Chris <(E-Mail Removed)>

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Evaluating static analysis and Dynamic analysis tools for C/C++ ssubbarayan C Programming 5 11-03-2009 12:50 AM
? ? crash dump analysis / memory check ? ? User Name Computer Support 6 01-11-2008 09:12 PM
Memory Profiler for Heap Analysis Moritz Wissenbach Java 14 06-20-2007 10:37 AM
python memory analysis Alexander Hoffmann Python 1 09-06-2004 09:26 PM
Understanding memory leaks in Java & start using HAT for further analysis qazmlp Java 3 01-07-2004 09:30 PM



Advertisments