Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Memory Leak

Reply
Thread Tools

Memory Leak

 
 
Guy
Guest
Posts: n/a
 
      02-24-2004
Hi

It might take me a little time to explain this but here goes.
Firstly I'm not using the latest upto date python releases so my first
plan is to try more upto date rels of python and win32 libs, as this
has fixed problems for me in the past (Don't like doing this as it
usally breaks stuff and sys admin are slow to update software,
although they do a cracking job, just covering my back.)

I've created a build script for my companies product, the product is
in c and can be built using makefiles and gmake and dsps if using
win32. The script seems to work ok, it builds by setting processes off
and then it returns the built libs exe to a network area.

The problems occure when I build more than one flavour of the product,
(there are 16 possible flavours and about 50 modules which get turned
into libs and exe's this means theres 800 processes to be set off one
after the other or if using a clean build 1600 process and thats just
for win32, to build one flavour it takes about 40 mins on my comp.)

When using the build script it keeps logs of what its built and
records all output from the process all these python objs and lists
are destroyed by using del to try and free up memory, this never seems
to have that much effect the memory just keeps getting used up until
usally the script or python crashes due to the lack of mem.

Sometimes for no reason the mem that python uses seem to purge itself
and start a fresh, I don't know why this is or how to control it.

I also don't think the del is working, not sure.

The script itself is made up of my code, python, python win32 libs and
a 3rd party module off the internet which controls the processes
(which has been of great help)

I just wondering if anybody can give me any ideas about whats going on
here, or maybe some memory purge commands or memory dianosics so I can
see which python objs are using all the mem.

TIA
Guy
 
Reply With Quote
 
 
 
 
Peter Hansen
Guest
Posts: n/a
 
      02-24-2004
Guy wrote:
>
> It might take me a little time to explain this but here goes.


Thanks for the details you did provide; it's more than most manage!

> Firstly I'm not using the latest upto date python releases so my first
> plan is to try more upto date rels of python and win32 libs, as this
> has fixed problems for me in the past (Don't like doing this as it
> usally breaks stuff and sys admin are slow to update software,
> although they do a cracking job, just covering my back.)


Definitely do that. There have been memory leaks in past versions
(and possibly are in the latest too, but it's perhaps less likely unless
you are using some of the newest features, which you obviously aren't).

I'd even suggest that doing much before you try that would be a waste.

> The problems occure when I build more than one flavour of the product,
> (there are 16 possible flavours and about 50 modules which get turned
> into libs and exe's this means theres 800 processes to be set off one
> after the other or if using a clean build 1600 process and thats just
> for win32, to build one flavour it takes about 40 mins on my comp.)
>
> When using the build script it keeps logs of what its built and
> records all output from the process all these python objs and lists
> are destroyed by using del to try and free up memory, this never seems
> to have that much effect the memory just keeps getting used up until
> usally the script or python crashes due to the lack of mem.


Memory leaks are extremely difficult to debug, perhaps more difficult
than any other problem except race conditions.

Thankfully, real memory leaks are very rare. I don't think you've
described enough to prove that you actually have a memory leak. What
you are describing sounds more like you're just using up all the
resources too quickly, which is more of a capacity (or design) problem
than anything else.

More importantly, you've got way too many variables involved to be
able to troubleshoot this, if it's really a memory leak. You absolutely
will have to simplify the setup, while keeping the problem occurring,
in order to solve this issue. If this is a memory leak involving Python,
you will be able to reproduce it with a program that involves nothing
but Python, and perhaps a small external dummy program that you launch
repeatedly in the other processes.

You shouldn't have to bother with "del" at all, since all that does is
release the name from the object it's bound to (or vice versa) and if
there are no other names bound to the object, Python will free the memory
anyway. (Note that it might not return the memory to the OS, however.
You can't force that to happen, either.) Doing a "del" followed by
reassigning the name to a new object is basically a waste of time.

Consider this. From the looks of things, you are doing so much that
is actually *outside* of the scope of Python that you might simply
be running out of memory elsewhere.

Also consider whether this might be due to memory fragmentation rather
than true leaking. You might need to analyze your pattern of memory
usage to see whether this could be the case.

-Peter
 
Reply With Quote
 
 
 
 
Cameron Laird
Guest
Posts: n/a
 
      02-24-2004
In article <(E-Mail Removed)>,
Peter Hansen <(E-Mail Removed)> wrote:
.
.
.
>Memory leaks are extremely difficult to debug, perhaps more difficult
>than any other problem except race conditions.
>
>Thankfully, real memory leaks are very rare. I don't think you've

.
[LOTS more sound advice]
.
.
Mostly, I just want to repeat Peter's sound counsel.

There's a LOT to say about memory errors; in fact, I'm even
trying to introduce the subject to C and C++ programmers <URL:
http://informit.com/isapi/product_id~{51BA92A2-20C7-4142-8707-FA7140CE4275}/session_id~{549AF63E-92F6-4EC1-B6B7-6A9147DF259E}/content/index.asp >
I'll pull out a few points present in Peter's description that
I think summarize the right attitude:
1. Look to yourself. Until you have strong evidence
to the contrary, assume that apparent memory pro-
blems are under your control, and are NOT an
intrinsic fault of Python.
2. The biggest return-on-investment you can reap is
to upgrade your version.
3. Next is to rationalize your own coding.
4. Next is to inspect code for memory faults. A
good developer can certify source, without re-
course to external tools. It's possible, even
practical, to learn all the basic categories
of memory error and be alert to their appear-
ance.
--

Cameron Laird <(E-Mail Removed)>
Business: http://www.Phaseit.net
 
Reply With Quote
 
Michael Hudson
Guest
Posts: n/a
 
      02-25-2004
http://www.velocityreviews.com/forums/(E-Mail Removed) (Guy) writes:

> Hi
>
> It might take me a little time to explain this but here goes.
> Firstly I'm not using the latest upto date python releases so my first
> plan is to try more upto date rels of python and win32 libs, as this
> has fixed problems for me in the past (Don't like doing this as it
> usally breaks stuff and sys admin are slow to update software,
> although they do a cracking job, just covering my back.)
>
> I've created a build script for my companies product, the product is
> in c and can be built using makefiles and gmake and dsps if using
> win32. The script seems to work ok, it builds by setting processes off
> and then it returns the built libs exe to a network area.
>
> The problems occure when I build more than one flavour of the product,
> (there are 16 possible flavours and about 50 modules which get turned
> into libs and exe's this means theres 800 processes to be set off one
> after the other or if using a clean build 1600 process and thats just
> for win32, to build one flavour it takes about 40 mins on my comp.)
>
> When using the build script it keeps logs of what its built and
> records all output from the process all these python objs and lists
> are destroyed by using del to try and free up memory, this never seems
> to have that much effect the memory just keeps getting used up until
> usally the script or python crashes due to the lack of mem.


Um, I don't think you understand what 'del' does.

For an oblique perspective on these issues, you might find reading the
messages mentioned in

http://starship.python.net/crew/mwh/hacks/pep310/

enlightening, particularly my last one. Quoting:

>> I guess I have a lispers approach to this: view memory as
>> infinite and assume the implementation does enough behind its
>> back to maintain this illusion so long as not too many objects
>> are actually live.

>
> If you seperate the memory and object concepts, an object can
> cease to exist even though memory is still allocated for it.


Well, this begs another question that I guess underlies all of
this: what does it mean for an object to be "alive"? Again, my
background is functional/lisp-like languages, and I'm stealing
this description more-or-less from Appel's "Modern Compiler
Implementation in ML":

An object is *live* if it will be used again by the executing
program.

At any given point of program execution, determining the set of
live objects can obviously be at least as hard as the halting
problem, so we settle for a weaker notion:

An object is *reachable* if it can be reached by traversing the
object graph from a given set of roots (in Python, things like
locals and the globals of all loaded modules).

The runtime pessimistically assumes that all reachable objects are
live (and reachable but not live objects can be a cause of memory
leaks in Python and other GCed languages).

From this point of view, you never actually /destroy/ objects, you
just forget about them, and a __del__ method is a side-effect of
dropping the last reference to an object -- a slightly unnatural
action to have a side effect!

This is obviously in contrast to C++, where you /do/ destroy
objects (or in the case of local variables, it is clear when the
object will be destroyed as a function of program flow).

I guess you are probably in the "reachable but not live" case of
memory leaks.

Instead of doing 'del alist' you might want to try 'del alist[:]' and
see if behaviour improves.

Cheers,
mwh

--
112. Computer Science is embarrassed by the computer.
-- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Memory leak even after deleting memory pointers from vector cham C++ 5 09-25-2008 10:30 AM
Leak or no leak ?? Richard Heathfield C Programming 4 07-10-2006 11:37 AM
Dynamic memory allocation and memory leak... s.subbarayan C Programming 10 03-22-2005 02:48 PM
Memory leak??? (top reporting high memory usage under Solaris) Mark Probert Ruby 4 02-09-2005 06:13 PM
Wireless Zero Configuration Memory Leak?? =?Utf-8?B?Umlja3NjaHVsdHox?= Wireless Networking 3 01-19-2005 11:26 PM



Advertisments