Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Reading just a few lines from a text file

Reply
Thread Tools

Reading just a few lines from a text file

 
 
tkpmep@hotmail.com
Guest
Posts: n/a
 
      08-23-2005
I have a text file with many hundreds of lines of data. The data of
interest to me, however, resides at the bottom of the file, in the last
20 lines. Right now, I read the entire file and discard the stuff I
don't need. I'd like to speed up my program by reading only the last 20
lines. How do I do this?

Thomas Philips

 
Reply With Quote
 
 
 
 
rafi
Guest
Posts: n/a
 
      08-23-2005
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> I have a text file with many hundreds of lines of data. The data of
> interest to me, however, resides at the bottom of the file, in the last
> 20 lines. Right now, I read the entire file and discard the stuff I
> don't need. I'd like to speed up my program by reading only the last 20
> lines. How do I do this?
>
> Thomas Philips
>


If you are using a Unix:

tail -20 file.txt

--
rafi

"Imagination is more important than knowledge."
(Albert Einstein)
 
Reply With Quote
 
 
 
 
draghuram@gmail.com
Guest
Posts: n/a
 
      08-23-2005

I just did strace on "tail -20 <filename>". Apparently, it does seek to
the end and reads enough data to cover 20 lines. I guess it is
calculating this "size" by counting 20 new lines .You may try to do the
same thing.

Thanks,
Raghu.

 
Reply With Quote
 
Paul McGuire
Guest
Posts: n/a
 
      08-23-2005
Are you sure this is really slowing down your program? "Many hundreds
of lines" is not nearly enough to start Python breathing hard. I have
been really impressed with just how quickly Python is able to do file
input and processing, zipping through whole megs of data in just
seconds.

How are you currently reading the file in? A character at a time?
That *will* be slow. Try file.readlines(), or xreadlines(), and spin
off the last 20 in the list.

-- Paul

 
Reply With Quote
 
tkpmep@hotmail.com
Guest
Posts: n/a
 
      08-23-2005
Right now my code reads as follows:

infile=file(FileName)
for line in reversed(infile.readlines()): #Search from the bottom up
if int(line.split()[0]) == MyDate:
Data= float(line.split()[-1])
break
infile.close()

I have to read about 10,000 files, each with data. I'm looking to speed
up each individual file open/close cycle.

Thomas Philips

 
Reply With Quote
 
Diez B. Roggisch
Guest
Posts: n/a
 
      08-23-2005
(E-Mail Removed) wrote:
> Right now my code reads as follows:
>
> infile=file(FileName)
> for line in reversed(infile.readlines()): #Search from the bottom up


Not sure if python does some tricks here - but for me that seems to be
uneccesary shuffling around of data. Better do

for line in reversed(infile.readlines()[:-10]):
...


Diez
 
Reply With Quote
 
John Machin
Guest
Posts: n/a
 
      08-23-2005
(E-Mail Removed) wrote:
> I have a text file with many hundreds of lines of data. The data of
> interest to me, however, resides at the bottom of the file, in the last
> 20 lines. Right now, I read the entire file and discard the stuff I
> don't need. I'd like to speed up my program by reading only the last 20
> lines. How do I do this?
>
> Thomas Philips
>


What discernible speed increase are you talking about? How long does it
take to read the "many hundreds" of lines?

For "many hundreds", IMHO it's not worth the bother, the complexity, the
documentation, ...

Just do this, it's about as fast as you'll get in pure python for a
smallish file:

def last_n_lines(filename, n):
return open(filename, 'r').readlines()[-n:]

For many hundreds of thousands of lines, one approach might be to open
the file in binary mode, seek to the end of the file, then loop reading
chunks backwards and unpacking the chunks until you've found 21 line
terminators. Or perhaps 20 line separators
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
XInclude a few lines of a text file? Boris XML 0 07-02-2008 10:58 PM
To delete few lines and add few lines at the end of a text file using c program Murali C++ 2 03-09-2006 04:45 PM
Help - just a few lines of code needed lory88@gmail.com Python 3 03-07-2006 02:08 PM
Help - just a few lines of code needed lory88@gmail.com Ruby 3 03-07-2006 06:52 AM
Help - just a few lines of code needed lory88@gmail.com C Programming 1 03-07-2006 04:01 AM



Advertisments