Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Iteration on file reading

Reply
Thread Tools

Iteration on file reading

 
 
Paul Watson
Guest
Posts: n/a
 
      10-02-2003
for line in sys.stdin:

Does this statement cause all of stdin to be read before the loop begins?

I may need to read several GB and I do not want to swamp the machine's
memory.


 
Reply With Quote
 
 
 
 
Andrew Dalke
Guest
Posts: n/a
 
      10-02-2003
Paul Watson
> for line in sys.stdin:
>
> Does this statement cause all of stdin to be read before the loop begins?


No. It will read a block of text at a time and break that block
into lines. This gives great performance and is scalable to
large files (so long as you can can afford to keep that extra
block around). However, it's lousy for interactive work.

Andrew
http://www.velocityreviews.com/forums/(E-Mail Removed)


 
Reply With Quote
 
 
 
 
Paul McGuire
Guest
Posts: n/a
 
      10-04-2003
Try a generator. This will just read a line at a time.
-- Paul

<code>
from sys import stdin

def lineReader( strm ):
while 1:
yield strm.readline().rstrip("\n")

for f in lineReader( stdin ):
print ">>> " + f
</code>

"Paul Watson" <(E-Mail Removed)> wrote in message
news:3f7ca9ea$(E-Mail Removed)...
> for line in sys.stdin:
>
> Does this statement cause all of stdin to be read before the loop begins?
>
> I may need to read several GB and I do not want to swamp the machine's
> memory.
>
>



 
Reply With Quote
 
Andrew Dalke
Guest
Posts: n/a
 
      10-04-2003
Paul McGuire:
> def lineReader( strm ):
> while 1:
> yield strm.readline().rstrip("\n")
>
> for f in lineReader( stdin ):
> print ">>> " + f


You can simplify that with the iter builtin.

for f in iter(stdin.readline, ""):
print ">>> " + f

(Hmm... maybe I should test it? Naaaaahhh.)

Andrew
(E-Mail Removed)


 
Reply With Quote
 
Just
Guest
Posts: n/a
 
      10-04-2003
In article <3f7ca9ea$(E-Mail Removed)>,
"Paul Watson" <(E-Mail Removed)> wrote:

> for line in sys.stdin:
>
> Does this statement cause all of stdin to be read before the loop begins?


Nope.

Just
 
Reply With Quote
 
Alex Martelli
Guest
Posts: n/a
 
      10-04-2003
Andrew Dalke wrote:

> Paul McGuire:
>> def lineReader( strm ):
>> while 1:
>> yield strm.readline().rstrip("\n")
>>
>> for f in lineReader( stdin ):
>> print ">>> " + f

>
> You can simplify that with the iter builtin.
>
> for f in iter(stdin.readline, ""):
> print ">>> " + f
>
> (Hmm... maybe I should test it? Naaaaahhh.)


There is a difference in behavior: the readline method
returns a line WITH a trailing \n, which then gets
printed, giving a "double-spaced" effect. Sure, you
can strip the \n in the loop body, but if you always
want a sequence of newline-stipped lines, that is
somewhat repetitious. If the use of readline is
mandated (i.e., no direct looping on the file for one
reason or another), my favourite way of expression is:

def linesof(somefile):
for line in iter(somefile.readline, ''):
yield line.rstrip('\n')

not as concise as either of the above, but, I think,
a wee little bit clearer.


Alex



 
Reply With Quote
 
Jeremy Fincher
Guest
Posts: n/a
 
      10-04-2003
"Paul Watson" <(E-Mail Removed)> wrote in message news:<3f7ca9ea$(E-Mail Removed)>...
> for line in sys.stdin:
>
> Does this statement cause all of stdin to be read before the loop begins?
>
> I may need to read several GB and I do not want to swamp the machine's
> memory.


Have you considered simply inputting this into an interactive
interpreter and seeing if it swamps the machine's memory?

Jeremy
 
Reply With Quote
 
Andrew Dalke
Guest
Posts: n/a
 
      10-04-2003
Alex:
> There is a difference in behavior: the readline method
> returns a line WITH a trailing \n, which then gets
> printed, giving a "double-spaced" effect. Sure, you
> can strip the \n in the loop body, ....


Quite true.

As it turns out, the OP wanted to know about

for line in sys.stdin:

The post to which I replied changed the spec to
remove the newline, but the main point was to
use a generator ... which could if desired to extra
work to get rid of the "\n". It could just have
easily converted everything to uppercase or done
rot13 conversion on the text.

My reply meant to point out that the iter builtin
can be used to turn a "function returns the next
object each time it's called and a sentinel when
it's done" into an iterable. I just left out the extra
work his code did since it wasn't needed by the OP.

Andrew
(E-Mail Removed)


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Iteration through File.file? misses entries for whichFile.file?(entry) == true Kyle Barbour Ruby 10 08-02-2010 08:55 PM
need a hash/iteration tutorial...text reading. Steven Demonnin Ruby 7 06-20-2009 12:41 PM
Struts - Problem with nested iteration or double iteration Rudi Java 5 10-01-2008 03:30 AM
iteration through a file of structs Dennis Schulz C Programming 2 05-08-2004 08:13 PM
open new file each loop iteration Danny Anderson C++ 0 01-21-2004 08:56 PM



Advertisments