windows mem leak
 

Does the Win32 port of Python have a memory leak? I have some code that
runs flawlessly on Linux, but bombs after a few hours on Windows. It's
threaded and uses a lot of memory.

Thanks!

Re: windows mem leak
 

Bob Smith wrote:
> Does the Win32 port of Python have a memory leak? I have some code that
> runs flawlessly on Linux, but bombs after a few hours on Windows. It's
> threaded and uses a lot of memory.

Let's see what you're missing:

1. platform specifics
2. versions of things involved
3. any sort of detail about the code
4. how you're noticing/measuring the problem
5. what "bombs" means
6. any mention that you've checked Sourceforge to see whether
a similar problem has been reported

There have been memory leaks in various past versions of Python,
and could easily be in the current version, but they're generally
rather specific in terms of the code that can trigger it. Once it
was doing some particular work with a socket, once it was trying to
append (or extend?) to an empty list, and so on.

There are also a few ways you could have written your application
to cause memory to leak as a result of your own code, not as a
result of Python's. And it's even possible that this would happen
only on one platform (though I'm trying hard to think of an example
and can't... maybe it's very unlikely.)

-Peter

Re: windows mem leak
 

Bob Smith wrote:

> Does the Win32 port of Python have a memory leak? I have some code that
> runs flawlessly on Linux, but bombs after a few hours on Windows. It's
> threaded and uses a lot of memory.
>
> Thanks!

Yes, that's a well-known problem. Code that runs with a few errors will
port without any trouble at all to Windows, but once it runs flawlessly
on Linux it starts to leak memory on Windows. The PSU suspects a plot in
Redmond, the basic details of which ar
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119

Re: windows mem leak
 

Steve Holden wrote:
> Bob Smith wrote:
>
>> Does the Win32 port of Python have a memory leak? I have some code
>> that runs flawlessly on Linux, but bombs after a few hours on Windows.
>> It's threaded and uses a lot of memory.
>>
>> Thanks!
>
>
> Yes, that's a well-known problem. Code that runs with a few errors will
> port without any trouble at all to Windows, but once it runs flawlessly
> on Linux it starts to leak memory on Windows. The PSU suspects a plot in
> Redmond, the basic details of which ar

Oh, the humor of it all ;)

Attached is the code. Run it yourself and see. You too Peter. Be gentle
with me, this was my first attempt with threads.

import os
import urllib
import socket
import time
import Queue
import threading

##########################################
# Network Section #
##########################################

socket.setdefaulttimeout(30)

networks = []
hosts = []
subnets = []

# Add the network 192.168.0 possibility to networks.
networks.append("192.168.0.")

# Generate and add networks 192.168.1-255 to networks.
n = 0
while n < 255:
n = n + 1
networks.append("192.168.%s." %(n))

# Generate and add hosts 1-254 to hosts.
for network in networks:
h = 1
# Add the n.n.n.0 host possibility to hosts.
hosts.append(network+str(h))
while h < 254:
h = h + 1
hosts.append(network+str(h))

# This should equal 65024 or 256 * 254
# because 256 possibilities are OK in the 3rd octet,
# but only 254 possibilities are OK in the 4th octet...
# we exclude 0 and 255.
print "There are", len(hosts), "total hosts (192.168.256*254) in the hosts list."

a = 0
b = 254

## Add the 192.168.0 net list to the subnets list.
subnets.append(hosts[0:254])
##print subnets
##print len(subnets)

## Add the 192.168.1-254 net lists to the subnets list.
for x in xrange(254):
a = a+254
b = b+254
subnets.append(hosts[a:b])
##print subnets
##print len(subnets)

## Add the 192.168.255 net list to the subnets list.
subnets.append(hosts[64770 :65024])
##print subnets[0]
print "There are", len(subnets), "class C network lists in the subnets list."

##########################################
# Queue Section #
##########################################

# Create a queue of urls to feed the threads
# Make it so that the queue only contains 256 items

nmap_queue = Queue.Queue(256)
for subnet in subnets:
nmap_queue.put(subnet)

###########################################
# Thread Section #
###########################################

class prac(threading.Thread):

def run(self):
net = nmap_queue.get()
for ip in net:
Y = os.popen('nmap -sT -p 80 -P0 -n %s' %ip)
data = Y.read()
Y.close()
if 'open' in data:
O = file('opened.txt', 'a')
print >> O, ip
O.close()
elif 'closed' in data:
C = file('closed.txt', 'a')
print >> C, ip
C.close()
elif 'filtered' in data:
F = file('filtered.txt' , 'a')
print >> F, ip
F.close()
else:
V = file('other.txt', 'a')
print >> V, data, ip
V.close()

threads = []
for i in xrange(256):
go = prac()
threads.append(go)
for thread in threads:
thread.start()
while threading.activeCount() > 1:
print str(threading.activeCount()), "threads running incl. main"
time.sleep(1)

Re: windows mem leak
 

On Sat, 08 Jan 2005 15:00:05 -0500, Steve Holden <steve@holdenweb.com>
wrote:

>> Does the Win32 port of Python have a memory leak? I have some code that
>> runs flawlessly on Linux, but bombs after a few hours on Windows. It's
>> threaded and uses a lot of memory.

>Yes, that's a well-known problem. Code that runs with a few errors will
>port without any trouble at all to Windows, but once it runs flawlessly
>on Linux it starts to leak memory on Windows. The PSU suspects a plot in
>Redmond, the basic details of which ar

You have my nomination for SOTW*. :-)

* skit of the week / short story of the week




--
It's a man's life in a Python Programming Association.

Re: windows mem leak
 

Bob Smith wrote:
> Attached is the code. Run it yourself and see. You too Peter. Be gentle
> with me, this was my first attempt with threads.

Thanks, Bob, and I will, but not before you answer some of my
questions.

I had good reasons to ask them, one of which is that I don't
feel like wasting my time if, for example, you are using an
older version of Python that *did* have a memory leak.

Python 2.2, for example, (if my memory serves me right) had
a leak in the first release which would have affected pretty
much all software which did network work.

The most important answers you can provide will be versions,
platform (pretty clearly Linux, but please confirm and give
version), and what "bombs" means and how you are measuring
the memory leak.

(I presume you're using a version of nmap that's compiled
for Windows XP then? It's certainly not standard. How have
you proven that it is not *that* program which is at fault?)

-Peter

Re: windows mem leak
 

Peter Hansen wrote:
> Bob Smith wrote:
>
>> Attached is the code. Run it yourself and see. You too Peter. Be
>> gentle with me, this was my first attempt with threads.
>
>
> Thanks, Bob, and I will, but not before you answer some of my
> questions.
>
> I had good reasons to ask them, one of which is that I don't
> feel like wasting my time if, for example, you are using an
> older version of Python that *did* have a memory leak.

2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 bit (Intel)]

>
> The most important answers you can provide will be versions,
> platform (pretty clearly Linux, but please confirm and give
> version), and what "bombs" means and how you are measuring
> the memory leak.

WinXP Home, Service Pack 2, AMD 1400MHz proc, 256MB Ram
Debian Linux Testing (2.4.28 vanilla Kernel) 3GHz P4 proc, 1.5GB Ram

>
> (I presume you're using a version of nmap that's compiled
> for Windows XP then?

Yes, I am.

> It's certainly not standard.

That's a matter of opinion. Nmap works fine on the WinXP machine.

> How have
> you proven that it is not *that* program which is at fault?)

I have not. All I know is that on WinXP, the program uses 100% CPU at
times and consumes more Ram than is available (the page file grows to
700 or 800MB). It runs OK for a few hours and then produces a 'not
enough resources' error. And, the machine is generally unuserable. On
Linux, it has no impact whatsoever on resources. Granted, the Linux
machine is much more robust, but one wouldn't expect this great a
difference. I can rewrite it so that it's pure Pyhton (no calling nmap)
if you think that would be a good idea. Perhaps that would at least
remove nmap from the equation.

I can run it if you like and take a screen shot of the error. You'll
have to give me a few hours though ;)

Re: windows mem leak
 

Bob Smith wrote:
> Peter Hansen wrote:
[snip details of Bob's platform]
> WinXP Home, Service Pack 2, AMD 1400MHz proc, 256MB Ram

That's not really much RAM for a WinXP box. Do you have
lots of physical memory available before running?

>> (I presume you're using a version of nmap that's compiled
>> for Windows XP then?
> Yes, I am.
>
>> It's certainly not standard.
> That's a matter of opinion. Nmap works fine on the WinXP machine.

Perhaps my use of "standard" was obscure, since it's definitely
not a matter of opinion, the way I intended it. What I meant
was "nmap is certainly not included in the standard Win XP
distribution".

>> How have
>> you proven that it is not *that* program which is at fault?)
>
> I have not. All I know is that on WinXP, the program uses 100% CPU at
> times and consumes more Ram than is available (the page file grows to
> 700 or 800MB). It runs OK for a few hours and then produces a 'not
> enough resources' error.

Is it certain that this memory is being consumed by the Python
process? I could imagine, for example, there being dozens of
new processes spawned with os.system. Does the Task Manager
back up the theory that this "python.exe" instance is the one
causing the trouble? Presumably it should be clear even before
the machine grows unusable. Note that you can select which
columns are shown in the "Processes" tab of the Task Manager
window, to get more detail on a given process. (The Performance
Monitor found under Control Panel, Administrative Tools can
be even more useful and precise.)


> And, the machine is generally unuserable. On
> Linux, it has no impact whatsoever on resources. Granted, the Linux
> machine is much more robust, but one wouldn't expect this great a
> difference. I can rewrite it so that it's pure Pyhton (no calling nmap)
> if you think that would be a good idea. Perhaps that would at least
> remove nmap from the equation.

A simpler test might be to change the nmap call to something
guaranteed benign, like a call to "dir", and try with that.
That's assuming you don't actually need the output of nmap
to reproduce the problem, which of course isn't sure. Still,
it would be an easy test, and might show a problem elsewhere.

> I can run it if you like and take a screen shot of the error. You'll
> have to give me a few hours though ;)

I trust that you are getting the error dialog you say you are. :-)
I have a feeling that the message by itself helps little, however,
and that you'll have to try a few different approaches to observe
the problem as it grows, via Task Manager or another tool, rather
than just trying to guess what happened after the machine is
already effective kaput.

-Peter

Re: windows mem leak
 

Bob Smith wrote:
> Peter Hansen wrote:
>> How have
>> you proven that it is not *that* program which is at fault?)

It would surprise me: even if it consumes much CPU-time, memory and
other resources, each instances returns all resources when it exits.

> I have not. All I know is that on WinXP, the program uses 100% CPU at
> times and consumes more Ram than is available (the page file grows to
> 700 or 800MB). It runs OK for a few hours and then produces a 'not
> enough resources' error. And, the machine is generally unuserable. On
> Linux, it has no impact whatsoever on resources. Granted, the Linux
> machine is much more robust, but one wouldn't expect this great a
> difference. I can rewrite it so that it's pure Pyhton (no calling nmap)
> if you think that would be a good idea. Perhaps that would at least
> remove nmap from the equation.

I wrote a very simple and small fake_nmap that just looks at the
IP-address and prints "open", "closed" or "filtered" to stdout. When I
run your python program (Python 2.4 on Windows XP, like you), the CPU is
utilized 100% (about half of it goes to csrss.exe whatever that may be);
about half of the CPU time is spent in the kernel. The system stays
usable (at least for now, it's been running for about 5 minutes now),
but memory use is increasing, slow but steadily.

The task manager shows, in addition to a number of fake_nmap.exe
processes, a number of cmd.exe processes. I don't understand where these
come from: I know os.system ()uses the shell, but isn't os.popen()
supposed to open the process directly? It seems there are a lot more
instances of cmd.exe than of fake_nmap.exe; no idea what that tells us.

Also, it takes quite some time before "256 threads running incl. main"
is printed the first time, so I think the system needs all that time to
create all the threads. It would be normal for memory use to keep
increasing untill all threads are created, but I'm fairly certain memory
use is still increasing now.

--
"Codito ergo sum"
Roel Schroeven

Re: windows mem leak
 

Roel Schroeven wrote:
>> Peter Hansen wrote:
>>> How have
>>> you proven that it is not *that* program which is at fault?)
>
> It would surprise me: even if it consumes much CPU-time, memory and
> other resources, each instances returns all resources when it exits.

I agree with that statement, but you assume that the program *is*
exiting. And your initial analysis with "fake_nmap" suggests
that, at least to the extent of having leftover cmd.exe's kicking
around, maybe it is not.

-Peter