Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   convert ftp.retrbinary to file object? - Python language lacks expression? (http://www.velocityreviews.com/forums/t341128-convert-ftp-retrbinary-to-file-object-python-language-lacks-expression.html)

Robert 02-03-2005 10:08 AM

convert ftp.retrbinary to file object? - Python language lacks expression?
 
I just tried to convert a (hugh size) ftp.retrbinary run into a
pseudo-file object with .read(bytes) method in order to not consume
500MB on a copy operation.

First I thought, its easy as usual with python using something like
'yield' or so.

Yet I didn't manage to do (without using threads or rewriting
'retrbinary')? Any ideas?

#### I tried a pattern like:
....
def open(self,ftppath,mode='rb'):
class FTPFile: #TODO
...
def iter_retr()
...
def callback(blk):
how-to-yield-from-here-to-iter_retr blk???
ftp.retrbinary("RETR %s" % relpath,callback)
def read(self, bytes=-1):
...
self.buf+=self.iter.next()
...
....

Martin Franklin 02-03-2005 11:07 AM

Re: convert ftp.retrbinary to file object? - Python language lacksexpression?
 
Robert wrote:
> I just tried to convert a (hugh size) ftp.retrbinary run into a
> pseudo-file object with .read(bytes) method in order to not consume
> 500MB on a copy operation.
>
> First I thought, its easy as usual with python using something like
> 'yield' or so.
>
> Yet I didn't manage to do (without using threads or rewriting
> 'retrbinary')? Any ideas?
>
> #### I tried a pattern like:
> ....
> def open(self,ftppath,mode='rb'):
> class FTPFile: #TODO
> ...
> def iter_retr()
> ...
> def callback(blk):
> how-to-yield-from-here-to-iter_retr blk???
> ftp.retrbinary("RETR %s" % relpath,callback)
> def read(self, bytes=-1):
> ...
> self.buf+=self.iter.next()
> ...
> ....



Hmmmm this is nearly there I think...:

import ftplib

class TransferAbort(Exception): pass

class FTPFile:
def __init__(self, server, filename):
self.server = server
self.filename = filename
self.offset = 0

def callback(self, data):
self.offset = self.offset + len(data)
self.data = data
## now quit the RETR command?
raise TransferAbort("stop right now")

def read(self, amount):
self.ftp = ftplib.FTP(self.server)
self.ftp.login()
try:
self.ftp.retrbinary("RETR %s" %self.filename, self.callback,
blocksize=amount,
rest=self.offset)
except TransferAbort:
return self.data


f = FTPFile("HOSTNAME", "FILENAME")

print f.read(24)
print f.read(24)


I open the ftp connection inside the read method as it caused an error
(on the second call to read) when I opened it in __init__ ???

HTH
Martin



Martin Franklin 02-03-2005 11:20 AM

Re: convert ftp.retrbinary to file object? - Python language lacksexpression?
 
Martin Franklin wrote:
> Robert wrote:
>
>> I just tried to convert a (hugh size) ftp.retrbinary run into a
>> pseudo-file object with .read(bytes) method in order to not consume
>> 500MB on a copy operation.


[snip]

>
>
> Hmmmm this is nearly there I think...:


whoops... spoke too soon..

>
> import ftplib
>
> class TransferAbort(Exception): pass
>
> class FTPFile:
> def __init__(self, server, filename):
> self.server = server
> self.filename = filename
> self.offset = 0
>
> def callback(self, data):
> self.offset = self.offset + len(data)
> self.data = data
> ## now quit the RETR command?
> raise TransferAbort("stop right now")
>
> def read(self, amount):
> self.ftp = ftplib.FTP(self.server)
> self.ftp.login()



I needed to insert a time.sleep(0.1) here as the connections were
falling over themselves - I guess testing with a blocksize of 24
is a little silly.


> try:
> self.ftp.retrbinary("RETR %s" %self.filename, self.callback,
> blocksize=amount,
> rest=self.offset)
> except TransferAbort:
> return self.data
>
>
> f = FTPFile("HOSTNAME", "FILENAME")
>
> print f.read(24)
> print f.read(24)
>


## new test...

f = FTPFile("HOSTNAME", "FILENAME")

while 1:
data = f.read(24)
if not data:
break
print data,


>
> I open the ftp connection inside the read method as it caused an error
> (on the second call to read) when I opened it in __init__ ???
>
> HTH
> Martin
>
>












Martin Franklin 02-03-2005 11:37 AM

Re: convert ftp.retrbinary to file object? - Python language lacksexpression?
 
Martin Franklin wrote:
> Martin Franklin wrote:
>
>> Robert wrote:
>>
>>> I just tried to convert a (hugh size) ftp.retrbinary run into a
>>> pseudo-file object with .read(bytes) method in order to not consume
>>> 500MB on a copy operation.

>
>
> [snip]
>
>>
>>
>> Hmmmm this is nearly there I think...:

>
>
> whoops... spoke too soon..



Trigger happy this morning...

>
>>
>> import ftplib
>>
>> class TransferAbort(Exception): pass
>>
>> class FTPFile:
>> def __init__(self, server, filename):
>> self.server = server
>> self.filename = filename
>> self.offset = 0
>>
>> def callback(self, data):
>> self.offset = self.offset + len(data)
>> self.data = data
>> ## now quit the RETR command?
>> raise TransferAbort("stop right now")
>>
>> def read(self, amount):
>> self.ftp = ftplib.FTP(self.server)
>> self.ftp.login()

>
>
>
> I needed to insert a time.sleep(0.1) here as the connections were
> falling over themselves - I guess testing with a blocksize of 24
> is a little silly.
>
>
>> try:
>> self.ftp.retrbinary("RETR %s" %self.filename, self.callback,
>> blocksize=amount,
>> rest=self.offset)
>> except TransferAbort:


also need to close the ftp connection here!

self.ftp.close()

>> return self.data
>>
>>
>> f = FTPFile("HOSTNAME", "FILENAME")
>>
>> print f.read(24)
>> print f.read(24)
>>

>
> ## new test...
>
> f = FTPFile("HOSTNAME", "FILENAME")
>
> while 1:
> data = f.read(24)
> if not data:
> break
> print data,
>
>
>>
>> I open the ftp connection inside the read method as it caused an error
>> (on the second call to read) when I opened it in __init__ ???
>>
>> HTH
>> Martin
>>
>>



Robert 02-03-2005 03:35 PM

Re: convert ftp.retrbinary to file object? - Python language lacksexpression?
 
That turns into periodic new RETR commands with offset. Think its more
an "odd" trick. I'd even prefer a threaded approach (thread puts the
blocks into a stack; a while ... yield generator loop in the main thread
serves the .read() function of the pseudo file object, which is my
wish). Yet such tricks are all kind of OS-level tricks with a lot of
overhead.

I wonder really, if the Python language itself can express an elegant
flat solution to turn the block delivering callback function into a
generator/.read(bytes) solution? I found no way.

(Looking over some Ruby stuff, Ruby seems to be able to do so from the
language. I am not really familiar to Ruby. I always felt Python to be
as complete - but much more clean. I became somewhat jealous ... :-) )

As the solution in my case has to stand many different file systems
compatibly ( file.read(bytes) function !) and also other FTPS & SFTP
classes with different retrbinary functions have to be compatible, I
cannot even make a simple FTP subclassed retrbinary without getting
really weired. Thus the existing .retrbinary with callback is the
"official interface in this game".



> >> def callback(self, data):
> >> self.offset = self.offset + len(data)
> >> self.data = data
> >> ## now quit the RETR command?
> >> raise TransferAbort("stop right now")


Steve Holden 02-03-2005 04:05 PM

Re: convert ftp.retrbinary to file object? - Python language lacksexpression?
 
Robert wrote:

> That turns into periodic new RETR commands with offset. Think its more
> an "odd" trick. I'd even prefer a threaded approach (thread puts the
> blocks into a stack; a while ... yield generator loop in the main thread
> serves the .read() function of the pseudo file object, which is my
> wish). Yet such tricks are all kind of OS-level tricks with a lot of
> overhead.
>
> I wonder really, if the Python language itself can express an elegant
> flat solution to turn the block delivering callback function into a
> generator/.read(bytes) solution? I found no way.
>

Don't know whether this would be helpful as a starting point, but a
while (hmm, some years ...) ago I wrote an example of how FTP could be
used as a file-like object. Look for ftpStream.py on

http://www.holdenweb.com/Python/

Of course, in those days files could do a bit less than they can now, so
there's no attempt to provide an iterator interface.

> (Looking over some Ruby stuff, Ruby seems to be able to do so from the
> language. I am not really familiar to Ruby. I always felt Python to be
> as complete - but much more clean. I became somewhat jealous ... :-) )
>
> As the solution in my case has to stand many different file systems
> compatibly ( file.read(bytes) function !) and also other FTPS & SFTP
> classes with different retrbinary functions have to be compatible, I
> cannot even make a simple FTP subclassed retrbinary without getting
> really weired. Thus the existing .retrbinary with callback is the
> "official interface in this game".
>

You will note that my code uses delegation to an FTP object rather than
inheritance. Maybe you would find that approach more fruitful for your
application.

regards
Steve
--
Meet the Python developers and your c.l.py favorites March 23-25
Come to PyCon DC 2005 http://www.pycon.org/
Steve Holden http://www.holdenweb.com/


All times are GMT. The time now is 09:16 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.