Re: Python garbage collector/memory manager behaving strangely
 

On Mon, 17 Sep 2012 10:28:34 +0800, "Jadhav, Alok"
<alok.jadhav@credit-suisse.com> declaimed the following in
gmane.comp.python.general:


> - As you have seen, the line separator is not '\n' but its '|\n'.
> Sometimes the data itself has '\n' characters in the middle of the line
> and only way to find true end of the line is that previous character
> should be a bar '|'. I was not able specify end of line using
> readlines() function, but I could do it using split() function.
> (One hack would be to readlines and combine them until I find '|\n'. is
> there a cleaner way to do this?)

Then chunk the file...

{PSEUDOCODE}

last = ""
while True:
chunk = fin.read(500000) #read 1/2 MB max
#and attach to left-overs
if not chunk: #EOF, process any left overs
processLine(last)
break #and exit

lines = (last + chunk).split("|\n")
last = lines[-1] #save "partial" line

for ln in lines[:-1]: #loop over rest
processLine(ln)

fin.close()
--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed@ix.netcom.com HTTP://wlfraed.home.netcom.com/