Re: Fast forward-backward (write-read)
On 10/23/2012 04:19 PM, David Hutto wrote:
> Whether this is fast enough, or not, I don't know:
well, the OP's original post started with
"I am working with some rather large data files (>100GB)..."
> filename = "data_file.txt"
> f = open(filename, 'r')
> forward = [line.rstrip('\n') for line in f.readlines()]
f.readlines() will be big(!) and have overhead... and forward results in
something again as big.
> backward = [line.rstrip('\n') for line in reversed(forward)]
and defining backward looks to me to require space to build backward and
So, let's see, at that point in time (building backward) you've got
probably somewhere close to 400-500Gb in memory.
My guess -- probably not so fast. Thrashing is sure to be a factor on
all but machines I'll never have a chance to work on.
> print forward, "\n\n", "********************\n\n", backward, "\n"
It's good to retain context.
|All times are GMT. The time now is 12:02 PM.|
Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.