Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > file.read() doesn't read the whole file

Reply
Thread Tools

file.read() doesn't read the whole file

 
 
Sreejith K
Guest
Posts: n/a
 
      03-20-2009
Hi,

>>> snapdir = './mango.txt_snaps'
>>> snap_cnt = 1
>>> block = 0
>>> import os
>>> os.chdir('/mnt/gfs_local')
>>> snap = open(snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block)),'r')
>>> snap.read()

'dfdfdgagdfgdf\ngdgfadgagadg\nagafg\n\nfs\nf\nsadf \n\nsdfsdfsadf\n'
>>> snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block))

'./mango.txt_snaps/snap1/0'

The above code works fine and it reads the whole file till EOF. But
when this method is used in a different scenario the file is not read
completely. I'll post the code that read only some part of the file...

self.snap = open(self.snapdir + '/snap%d/%d' % (self.snap_cnt,
block),'r') ## opens /mnt/gfs_local/mango.txt_snaps/snap1/0
self.snap.seek(off%4096) ## seeks to 0 in this case
bend = 4096-(off%4096) ## 4096 in this case
if length-bend <= 0: ## true in this case as length is 4096
tf.writelines("returned \n")
data = self.snap.read(length)
self.snap.close()
break

the output data is supposed to read the whole fie but it only reads a
part of it. Why is it encountering an early EOF ?
 
Reply With Quote
 
 
 
 
R. David Murray
Guest
Posts: n/a
 
      03-20-2009
Sreejith K <(E-Mail Removed)> wrote:
> Hi,
>
> >>> snapdir = './mango.txt_snaps'
> >>> snap_cnt = 1
> >>> block = 0
> >>> import os
> >>> os.chdir('/mnt/gfs_local')
> >>> snap = open(snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block)),'r')
> >>> snap.read()

> 'dfdfdgagdfgdf\ngdgfadgagadg\nagafg\n\nfs\nf\nsadf \n\nsdfsdfsadf\n'
> >>> snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block))

> './mango.txt_snaps/snap1/0'
>
> The above code works fine and it reads the whole file till EOF. But
> when this method is used in a different scenario the file is not read
> completely. I'll post the code that read only some part of the file...
>
> self.snap = open(self.snapdir + '/snap%d/%d' % (self.snap_cnt,
> block),'r') ## opens /mnt/gfs_local/mango.txt_snaps/snap1/0
> self.snap.seek(off%4096) ## seeks to 0 in this case
> bend = 4096-(off%4096) ## 4096 in this case
> if length-bend <= 0: ## true in this case as length is 4096
> tf.writelines("returned \n")
> data = self.snap.read(length)
> self.snap.close()
> break
>
> the output data is supposed to read the whole fie but it only reads a
> part of it. Why is it encountering an early EOF ?


It's not. In the second case you told it to read only 4096 bytes. You
might want to read the docs for the 'read' method, paying particular
attention to the optional argument and its meaning.

--
R. David Murray http://www.bitdance.com

 
Reply With Quote
 
 
 
 
Sreejith K
Guest
Posts: n/a
 
      03-20-2009
On Mar 20, 4:43*pm, "R. David Murray" <(E-Mail Removed)> wrote:
> Sreejith K <(E-Mail Removed)> wrote:
> > Hi,

>
> > >>> snapdir = './mango.txt_snaps'
> > >>> snap_cnt = 1
> > >>> block = 0
> > >>> import os
> > >>> os.chdir('/mnt/gfs_local')
> > >>> snap = open(snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block)),'r')
> > >>> snap.read()

> > 'dfdfdgagdfgdf\ngdgfadgagadg\nagafg\n\nfs\nf\nsadf \n\nsdfsdfsadf\n'
> > >>> snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block))

> > './mango.txt_snaps/snap1/0'

>
> > The above code works fine and it reads the whole file till EOF. But
> > when this method is used in a different scenario the file is not read
> > completely. I'll post the code that read only some part of the file...

>
> > self.snap = open(self.snapdir + '/snap%d/%d' % (self.snap_cnt,
> > block),'r') ## opens /mnt/gfs_local/mango.txt_snaps/snap1/0
> > self.snap.seek(off%4096) ## seeks to 0 in this case
> > bend = 4096-(off%4096) ## 4096 in this case
> > if length-bend <= 0: * *## true in this case as length is 4096
> > * *tf.writelines("returned \n")
> > * *data = self.snap.read(length)
> > * *self.snap.close()
> > * *break

>
> > the output data is supposed to read the whole fie but it only reads a
> > part of it. Why is it encountering an early EOF ?

>
> It's not. *In the second case you told it to read only 4096 bytes. *You
> might want to read the docs for the 'read' method, paying particular
> attention to the optional argument and its meaning.
>
> --
> R. David Murray * * * * *http://www.bitdance.com


I'm using the above codes in a pthon-fuse's file class's read
function. The offset and length are 0 and 4096 respectively for my
test inputs. When I open a file and read the 4096 bytes from offset,
only a few lines are printed, not the whole file. Actually the file is
only a few bytes. But when I tried reading from the Interactive mode
of python it gave the whole file.

Is there any problem using read() method in fuse-python ?

Also statements like break and continue behaves weirdly in fuse
functions. Any help is appreciated....
 
Reply With Quote
 
R. David Murray
Guest
Posts: n/a
 
      03-20-2009
On Fri, 20 Mar 2009 at 07:09, Sreejith K wrote:
> On Mar 20, 4:43*pm, "R. David Murray" <(E-Mail Removed)> wrote:
>> Sreejith K <(E-Mail Removed)> wrote:
>>> Hi,

>>
>>>>>> snapdir = './mango.txt_snaps'
>>>>>> snap_cnt = 1
>>>>>> block = 0
>>>>>> import os
>>>>>> os.chdir('/mnt/gfs_local')
>>>>>> snap = open(snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block)),'r')
>>>>>> snap.read()
>>> 'dfdfdgagdfgdf\ngdgfadgagadg\nagafg\n\nfs\nf\nsadf \n\nsdfsdfsadf\n'
>>>>>> snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block))
>>> './mango.txt_snaps/snap1/0'

>>
>>> The above code works fine and it reads the wholefiletill EOF. But
>>> when this method is used in a different scenario thefileis notread
>>> completely. I'll post the code thatreadonly some part of thefile...

>>
>>> self.snap = open(self.snapdir + '/snap%d/%d' % (self.snap_cnt,
>>> block),'r') ## opens /mnt/gfs_local/mango.txt_snaps/snap1/0
>>> self.snap.seek(off%4096) ## seeks to 0 in this case
>>> bend = 4096-(off%4096) ## 4096 in this case
>>> if length-bend <= 0: * *## true in this case as length is 4096
>>> * *tf.writelines("returned \n")
>>> * *data = self.snap.read(length)
>>> * *self.snap.close()
>>> * *break

>>
>>> the output data is supposed toreadthe whole fie but it only reads a
>>> part of it. Why is it encountering an early EOF ?

>>
>> It's not. *In the second case you told it toreadonly 4096 bytes. *You
>> might want toreadthe docs for the 'read' method, paying particular
>> attention to the optional argument and its meaning.
>>
>> --
>> R. David Murray * * * * *http://www.bitdance.com

>
> Thanks for the reply,
> Actually the file is only few bytes and file.read() and file.read
> (4096) will give the same result, i.e the whole file. But in my case
> its not happening (using it in python-fuse file class).. Any other
> ideas ?


Not offhand. If it were me I'd start playing with parameters and moving
things around, trying to find additional clues. See if calling read
without the argument in the second case works, start stripping the second
case down until it starts working (even if wrongly for the ultimate goal),
and things like that.

--
R. David Murray http://www.bitdance.com
 
Reply With Quote
 
Terry Reedy
Guest
Posts: n/a
 
      03-20-2009
Sreejith K wrote:

> I'm using the above codes in a pthon-fuse's file class's read
> function. The offset and length are 0 and 4096 respectively for my
> test inputs. When I open a file and read the 4096 bytes from offset,
> only a few lines are printed, not the whole file. Actually the file is
> only a few bytes. But when I tried reading from the Interactive mode
> of python it gave the whole file.
>
> Is there any problem using read() method in fuse-python ?
>
> Also statements like break and continue behaves weirdly in fuse
> functions. Any help is appreciated....


I had never heard of fuse until today. I get the impression that
working with fuse is not the same as working with the stock os
filesystem, so you might have mentioned this from the beginning. I
suggest you direct these questions to python-fuse and fuse-python folk.

 
Reply With Quote
 
I V
Guest
Posts: n/a
 
      03-20-2009
On Fri, 20 Mar 2009 07:03:35 -0700, Sreejith K wrote:
> I'm using the above codes in a pthon-fuse's file class's read function.
> The offset and length are 0 and 4096 respectively for my test inputs.
> When I open a file and read the 4096 bytes from offset, only a few lines
> are printed, not the whole file. Actually the file is only a few bytes.
> But when I tried reading from the Interactive mode of python it gave the
> whole file.


Your example doesn't show what you are doing with "data" after you've
read it. Presumably you're outputting it somehow, which is where you see
that it doesn't contain the whole file. But are you sure the problem is
in the reading, and not in the outputting? Could you show the section of
the code where you output "data"?
 
Reply With Quote
 
Sreejith K
Guest
Posts: n/a
 
      03-21-2009
On Mar 21, 12:58*am, I V <(E-Mail Removed)> wrote:
> On Fri, 20 Mar 2009 07:03:35 -0700, Sreejith K wrote:
> > I'm using the above codes in a pthon-fuse's file class's read function.
> > The offset and length are 0 and 4096 respectively for my test inputs.
> > When I open a file and read the 4096 bytes from offset, only a few lines
> > are printed, not the whole file. Actually the file is only a few bytes.
> > But when I tried reading from the Interactive mode of python it gave the
> > whole file.

>
> Your example doesn't show what you are doing with "data" after you've
> read it. Presumably you're outputting it somehow, which is where you see
> that it doesn't contain the whole file. But are you sure the problem is
> in the reading, and not in the outputting? Could you show the section of
> the code where you output "data"?

Thanks,

I think there is no use in showing the whole code (which is a
filesystem implementation using python-fuse), but if you want it I'll
post it, see the next post

Forget about the 'data' variable (I'm just returning it at the end of
the function). But even if I'm returning self.snap.read(length), it
doesn't read length (if length is larger than the size of the file
then till EOF) bytes from the file.

But this only happens when the second code samples (they're used in
fuse-python classes) and not in the first example. In the normal
interactive python shell the same procedure returns the whole file
(using read() or read(4096) which is a particular case)...

I'm sure, In my case, self.snap is opening the file ./mango.txt_snaps/
snap1/0 and off and length variables are 0 and 4096 respectively. The
file ./mango.txt_snaps/snap1/0 is only 56 bytes with 9 lines of text.
But either read(4096)/read() in the second example reads/returns only
7 lines. But this happens only in when using it in a fuse-python file
class's read function. In normal interactive mode it reads the whole
file. Hope you understand the problem now. I'm very sorry that I
couldn't explain the problem very well.
 
Reply With Quote
 
Sreejith K
Guest
Posts: n/a
 
      03-21-2009
class MedusaFile(object):
def __init__(self, path, flags, *mode):
global METHOD
global NORMAL
global SNAP
global FRESH_SNAP
self.path = path
tf.writelines("File initiating..\n")
self.file = os.fdopen(os.open("." + path, flags, *mode),flag2mode
(flags))
self.fd = self.file.fileno()
curdir = GetDirPath('.' + path)
self.snapdir = '.' + path + '_snaps'
self.snap_cnt = 0
if os.path.exists(self.snapdir):
self.snap_cnt = len(os.listdir(self.snapdir))
METHOD = SNAP
elif FRESH_SNAP == 1:
self.snap_cnt += 1
os.mkdir(self.snapdir)
os.mkdir(self.snapdir+'/snap%s' % repr(self.snap_cnt))
METHOD = SNAP
FRESH_SNAP = 0
tf.writelines("File initiated..\n")
def read(self, length, offset):
global METHOD
global NORMAL
global SNAP
tf.writelines("Read length: %d offset: %d\n" % (length,offset))
blk_num = offset/4096
no_blks = length/4096 + 1
blocks = []
block_read = False
## Read form the Base File (Snap 1)
if METHOD == NORMAL:
self.file.seek(offset)
tf.writelines("Normal read\n")
return self.file.read(length)
## Read blocks from the snapshots
else:
snap_list = range(self.snap_cnt)
rev_snap_list = snap_list
rev_snap_list.reverse()
for i in snap_list:
blocks.append(os.listdir(self.snapdir+'/snap%s' % repr(i+1))) ##
list of blocks in the current snapshot
off = offset
bend = 0
data = ''
for block in range(blk_num,blk_num+no_blks): ## loop through the
blocks which are to be read
for i in rev_snap_list: ## loop through snapshots (starting from
latest snap)
tf.writelines('Snapshot %d opened..\n' % i)
if repr(block) in blocks[i]: ## Check if it is in the snapshot
tf.writelines("Snap read\n")
self.snap = open(self.snapdir + '/snap%d/%d' % (i+1,
block),'r')
self.snap.seek(off%4096)
bend = 4096-(off%4096)
block_read = True
if length-bend <= 0: ## if only a part of block file is to be
read (i.e. not till the end of block file)
tf.writelines("Partial read from snap \n")
data = self.snap.read(length)
self.snap.close()
break
tf.writelines("Full block read from snap\n")
data += self.snap.read(bend)
length -= bend
off = 4096
self.snap.close()
break
tf.writelines("Block not in snap\n")
if block_read:
block_read = False
continue
## otherwise the block should be in the base file itself. So,
read from there
tf.writelines("Reading from Base File\n")
self.file.seek(block*4096 + off%4096)
bend = 4096-(off%4096)
if length-bend <= 0: ## if only a part of a block is to be read
(not till the end of the block)
data = self.file.read(length)
break
data += self.file.read(bend)
length -= bend
off = 4096
return data

This is the filesystem class for files. Whenever a read occurs an
instance is created and read function is called. In my example when
accessing a file named 'mango.txt' it checks for mango.txt_snaps/snap1
dirctory and open file '0' as self.snap. But the read() returns (i.e
data) a small part....

Almost all the code worked weird in this example. Apart from read(),
the break and continue also works weird. When opening the file
'mango.txt' the following output is written by tf (an output file).
Here METHOD is not NORMAL, self.snap_cnt is 1, blocks is [['0']]

File initiating..
File initiated..
Read length: 4096 offset: 0
Snapshot 0 opened..
Snap read
Partial read from snap
Snapshot 0 opened..
Block not in snap
Reading from Base File

See the weirdness of continue and break here ?(loop was supposed to
loop only once as rev_snap_list contains 0 only)
 
Reply With Quote
 
R. David Murray
Guest
Posts: n/a
 
      03-21-2009
Sreejith K <(E-Mail Removed)> wrote:
> tf.writelines("Reading from Base File\n")
> self.file.seek(block*4096 + off%4096)
> bend = 4096-(off%4096)
> if length-bend <= 0: ## if only a part of a block is to be read
> (not till the end of the block)
> data = self.file.read(length)
> break
> data += self.file.read(bend)
> length -= bend
> off = 4096
> return data
>
> This is the filesystem class for files. Whenever a read occurs an
> instance is created and read function is called. In my example when
> accessing a file named 'mango.txt' it checks for mango.txt_snaps/snap1
> dirctory and open file '0' as self.snap. But the read() returns (i.e
> data) a small part....
>
> Almost all the code worked weird in this example. Apart from read(),
> the break and continue also works weird. When opening the file
> 'mango.txt' the following output is written by tf (an output file).
> Here METHOD is not NORMAL, self.snap_cnt is 1, blocks is [['0']]
>
> File initiating..
> File initiated..
> Read length: 4096 offset: 0
> Snapshot 0 opened..
> Snap read
> Partial read from snap
> Snapshot 0 opened..
> Block not in snap
> Reading from Base File
>
> See the weirdness of continue and break here ?(loop was supposed to
> loop only once as rev_snap_list contains 0 only)


I'm not going to look at all this code now...it looks way too complicated
and in need of some serious refactoring

But a couple of on-point comments:

How do you know rev_snap_list contains only 0? You didn't log it.

Same for the read. How do you know the read didn't read the whole
file? You didn't log it.

Both your statements might be true, but until you show the logging
output proving it, you don't _know_ that your assumptions are true.

--
R. David Murray http://www.bitdance.com

 
Reply With Quote
 
Sreejith K
Guest
Posts: n/a
 
      03-21-2009
On Mar 21, 10:54*am, "R. David Murray" <(E-Mail Removed)> wrote:
> Sreejith K <(E-Mail Removed)> wrote:
> > * * * * * * * * * * * * * * * * * *tf.writelines("Reading from Base File\n")
> > * * * * * * * * * * * * * * * * * *self.file.seek(block*4096 + off%4096)
> > * * * * * * * * * * * * * * * * * *bend = 4096-(off%4096)
> > * * * * * * * * * * * * * * * * * *if length-bend <= 0: ## if only a part of a block is to be read
> > (not till the end of the block)
> > * * * * * * * * * * * * * * * * * * * * * *data = self.file.read(length)
> > * * * * * * * * * * * * * * * * * * * * * *break
> > * * * * * * * * * * * * * * * * * *data += self.file.read(bend)
> > * * * * * * * * * * * * * * * * * *length -= bend
> > * * * * * * * * * * * * * * * * * *off = 4096
> > * * * * * * * * * * * * * *return data

>
> > This is the filesystem class for files. Whenever a read occurs an
> > instance is created and read function is called. In my example when
> > accessing a file named 'mango.txt' it checks for mango.txt_snaps/snap1
> > dirctory and open file '0' as self.snap. But the read() returns (i.e
> > data) a small part....

>
> > Almost all the code worked weird in this example. Apart from read(),
> > the break and continue also works weird. When opening the file
> > 'mango.txt' the following output is written by tf (an output file).
> > Here METHOD is not NORMAL, self.snap_cnt is 1, blocks is [['0']]

>
> > File initiating..
> > File initiated..
> > Read length: 4096 offset: 0
> > Snapshot 0 opened..
> > Snap read
> > Partial read from snap
> > Snapshot 0 opened..
> > Block not in snap
> > Reading from Base File

>
> > See the weirdness of continue and break here ?(loop was supposed to
> > loop only once as rev_snap_list contains 0 only)

>
> I'm not going to look at all this code now...it looks way too complicated
> and in need of some serious refactoring
>
> But a couple of on-point comments:
>
> How do you know rev_snap_list contains only 0? *You didn't log it.
>
> Same for the read. *How do you know the read didn't read the whole
> file? *You didn't log it.
>
> Both your statements might be true, but until you show the logging
> output proving it, you don't _know_ that your assumptions are true.
>
> --
> R. David Murray * * * * *http://www.bitdance.com


Thanks for your comments.

As I said, self.snap_cnt is 1 then

>>> snap_list = range(1)
>>> rev_snap_list = snap_list
>>> rev_snap_list

[0]
>>> rev_snap_list.reverse()
>>> rev_snap_list

[0]

So I thought there is no need to log it. In the above code, as break
and continue doesn't work as I expected the read() is called twice
("Partial read from snap" and "reading from base file") the output is
a combination of the 2 reads (so it can't be read using 'less'). But
the expected result was "Partial read from snap" i.e read(4096) and
break from the loop. Instead after this break, the loop is again
started ("Snapshot 0 opened.. "). I have no idea what went wrong here.
But when I comment out block_read = False before the continue
statement the log displays the following and the file is read
partially....

File initiating..
File initiated..
Read length: 4096 offset: 0
Snapshot 0 opened..
Snap read
Partial read from snap
Snapshot 0 opened..
Block not in snap
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
File.read(fname) vs. File.read(fname,File.size(fname)) Alex Dowad Ruby 4 05-01-2010 08:20 AM
read() does not read whole file in activepython/DOS Holger Python 1 11-07-2008 02:35 PM
Check if program finished to read a whole file lovecreatesbea...@gmail.com C++ 5 01-23-2007 05:39 PM
*WITHOUT* using: ValidateRequest="False" for the whole page (or my whole site).... \A_Michigan_User\ ASP .Net 2 08-21-2006 02:13 PM
After read a whole OS written by C, someone asks `What is the advantage of C' lovecreatesbeauty C Programming 22 07-18-2006 08:24 AM



Advertisments