Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > ftplib - Did the whole file get sent?

Reply
Thread Tools

ftplib - Did the whole file get sent?

 
 
Sean DiZazzo
Guest
Posts: n/a
 
      10-23-2010
Hi,

I have some scripts that send files via ftplib to a client's ftp
site. The scripts have generally worked great for a few years.
Recently, the client complained that only part of an important file
made it to their server. My boss got this complaint and brought it to
my attention.

The first thing I did was track down the specific file transfer in my
logs. My log showed a success, I told my boss that, but he wasn't
satisfied with my response. He began asking if there is a record of
the file transfer ack and number of bytes sent for this transfer. I'm
not keeping a record of that...only success or failure (and some
output)

How can I assure him (and the client) that the transfer completed
successfully like my log shows? I'm using code similar to the
following:

try:
ftp = ftplib.FTP(host)
ftp.login(user, pass)
ftp.storbinary("STOR " + destfile, open(f.path, 'rb'))
# log this as success
except:
# log this as an error

Is ftplib reliable enough to say that if an exception is not thrown,
that the file was transferred in full?

Python 2.4.3 (#1, Sep 17 2008, 16:07:0
Red Hat Enterprise Linux Server release 5.3 (Tikanga)

Thanks for your thoughts.

~Sean

 
Reply With Quote
 
 
 
 
Steven D'Aprano
Guest
Posts: n/a
 
      10-23-2010
On Fri, 22 Oct 2010 22:03:38 -0700, Sean DiZazzo wrote:

> How can I assure him (and the client) that the transfer completed
> successfully like my log shows?


"It has worked well for many years, there are no reported bugs in the ftp
code, and the logs show the file was transferred correctly. Unless you
can reproduce the failure, there's no evidence that my code is to blame
for the failure. How do you know that the original file wasn't corrupted
before the transfer? Or that the copy wasn't corrupted after the
transfer? Does it copy correctly if you try again? Can you give me a copy
of the file for testing?"



[...]
> Is ftplib reliable enough to say that if an exception is not thrown,
> that the file was transferred in full?


Python 2.4 is pretty old. Have you checked the bug tracker to see if
there are any reported bugs in ftplib that might be relevant? Or the
What's New for 2.5, 2.6 and 2.7? The source code for ftplib seems fairly
straightforward to me -- if there was an error, I can't see that it could
have been suppressed.

But really, unless you can reproduce the error, I'd say the error lies
elsewhere.


--
Steven
 
Reply With Quote
 
 
 
 
Sean DiZazzo
Guest
Posts: n/a
 
      10-23-2010
On Oct 22, 10:48*pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.au> wrote:
> On Fri, 22 Oct 2010 22:03:38 -0700, Sean DiZazzo wrote:
> > How can I assure him (and the client) that the transfer completed
> > successfully like my log shows?

>
> "It has worked well for many years, there are no reported bugs in the ftp
> code
> [...]


Thanks for your advice Steven. I agree with you,and did take that
approach to start with. Then the request for the actual FTP "ack"
caught me off guard. I had to break out Wireshark and run a few tests
just to make sure I knew exactly what I was talking about.

I think I will try to explain that asking for the "ack" is not really
a valid request. Something like this:

"Technically, these messages are used only on the lowest level of the
FTP protocol itself. Any client or library implementing FTP would be
sending these messages under the covers...in this case I think its
done in the socket library. It is possible that there is a bug in the
Python FTP library, just like it's possible there is a bug in any
other FTP client. Considering how long this library has been around
(~15-20 years), and how often it is used, it is very unlikely that a
bug causing a partial transfer but showing a success has managed to
stick around for so long."

Does that make sense?

> > Is ftplib reliable enough to say that if an exception is not thrown,
> > that the file was transferred in full?

>
> Python 2.4 is pretty old. Have you checked the bug tracker to see if
> there are any reported bugs in ftplib that might be relevant? Or the
> What's New for 2.5, 2.6 and 2.7? The source code for ftplib seems fairly
> straightforward to me -- if there was an error, I can't see that it could
> have been suppressed.
>
> But really, unless you can reproduce the error, I'd say the error lies
> elsewhere.
>
> --
> Steven


I'll check bugs and whats new before sending any response. The more I
think about this, I am beginning to think that he is just trying to
find someone to blame for a problem, and has chosen me.

Thanks again.

~Sean
 
Reply With Quote
 
Brendan
Guest
Posts: n/a
 
      10-25-2010
On Oct 23, 1:03*pm, Sean DiZazzo <(E-Mail Removed)> wrote:
> On Oct 22, 10:48*pm, Steven D'Aprano <st...@REMOVE-THIS-
>
> cybersource.com.au> wrote:
> > On Fri, 22 Oct 2010 22:03:38 -0700, Sean DiZazzo wrote:
> > > How can I assure him (and the client) that the transfer completed
> > > successfully like my log shows?

>
> > "It has worked well for many years, there are no reported bugs in the ftp
> > code
> > [...]

>
> Thanks for your advice Steven. *I agree with you,and did take that
> approach to start with. *Then the request for the actual FTP "ack"
> caught me off guard. *I had to break out Wireshark and run a few tests
> just to make sure I knew exactly what I was talking about.
>
> I think I will try to explain that asking for the "ack" is not really
> a valid request. *Something like this:
>
> "Technically, these messages are used only on the lowest level of the
> FTP protocol itself. *Any client or library implementing FTP would be
> sending these messages under the covers...in this case I think its
> done in the socket library. *It is possible that there is a bug in the
> Python FTP library, just like it's possible there is a bug in any
> other FTP client. *Considering how long this library has been around
> (~15-20 years), and how often it is used, it is very unlikely that a
> bug causing a partial transfer but showing a success has managed to
> stick around for so long."
>
> Does that make sense?
>
> > > Is ftplib reliable enough to say that if an exception is not thrown,
> > > that the file was transferred in full?

>
> > Python 2.4 is pretty old. Have you checked the bug tracker to see if
> > there are any reported bugs in ftplib that might be relevant? Or the
> > What's New for 2.5, 2.6 and 2.7? The source code for ftplib seems fairly
> > straightforward to me -- if there was an error, I can't see that it could
> > have been suppressed.

>
> > But really, unless you can reproduce the error, I'd say the error lies
> > elsewhere.

>
> > --
> > Steven

>
> I'll check bugs and whats new before sending any response. *The more I
> think about this, I am beginning to think that he is just trying to
> find someone to blame for a problem, and has chosen me.
>
> Thanks again.
>
> ~Sean


Your boss is both moron and wanker.
 
Reply With Quote
 
sjm
Guest
Posts: n/a
 
      10-25-2010
On Oct 23, 2:03*am, Sean DiZazzo <(E-Mail Removed)> wrote:

I follow every ftp put (STOR) with a dir command. Then if the
recipient claims that they never got it (or did not get all of it), I
have evidence that they did and that their byte count is the same as
mine.

This does not entirely guarantee that the ftp was perfect but it goes
a long way. It also provides useful information if there is a
problem.

HTH,
SJM

 
Reply With Quote
 
John Nagle
Guest
Posts: n/a
 
      10-25-2010
On 10/22/2010 10:03 PM, Sean DiZazzo wrote:
> Hi,
>
> I have some scripts that send files via ftplib to a client's ftp
> site. The scripts have generally worked great for a few years.
> Recently, the client complained that only part of an important file
> made it to their server. My boss got this complaint and brought it to
> my attention.
>
> The first thing I did was track down the specific file transfer in my
> logs. My log showed a success, I told my boss that, but he wasn't
> satisfied with my response. He began asking if there is a record of
> the file transfer ack and number of bytes sent for this transfer. I'm
> not keeping a record of that...only success or failure (and some
> output)
>
> How can I assure him (and the client) that the transfer completed
> successfully like my log shows? I'm using code similar to the
> following:
>
> try:
> ftp = ftplib.FTP(host)
> ftp.login(user, pass)
> ftp.storbinary("STOR " + destfile, open(f.path, 'rb'))
> # log this as success
> except:
> # log this as an error
>
> Is ftplib reliable enough to say that if an exception is not thrown,
> that the file was transferred in full?


No.

This was for years an outstanding problem with FTP under Windows.
See "http://www.fourmilab.ch/documents/corrupted_downloads/"
And
"http://us.generation-nt.com/answer/incomplete-ftp-upload-under-windows-xp-help-139017881.html"
And "http://winscp.net/forum/viewtopic.php?t=6458". Many FTP
implementations have botched this. TCP has all the machinery to
guarantee that both ends know the transfer completed
properly, but it's often misused.

Looking at the Python source, it doesn't look good. The "ftplib"
module does sending by calling sock_sendall in "socketmodule.c".
That does an OS-level "send", and once everything has been sent,
returns.

But an OS-level socket send returns when the data is queued for
sending, not when it is delivered. Only when the socket is closed,
and the close status checked, do you know if the data was delivered.
There's a final TCP close handshake that occurs when close has
been called at both ends, and only when it completes successfully
do you know that the data has been delivered.

At the socket level, this is performed by "shutdown" (which
closes the connection and returns the proper network status
information), or by "close" (which forces a shutdown but doesn't
return status).

Look at sock_close in "socketmodule.c". Note that it ignores the
return status on close, always returns None, and never raises an
exception. As the Linux manual page for "close" says:
"Not checking the return value of close() is a common but nevertheless
serious programming error. It is quite possible that errors on a
previous write(2) operation are first reported at the final close(). Not
checking the return value when closing the file may lead to silent loss
of data."

"ftplib", in "storlines" and "storbinary", calls "close"
without calling "shutdown" first. So if the other end disconnects
after all data has been queued but not received, the sender will
never know. FAIL.

So there's your bug.

John Nagle
 
Reply With Quote
 
Lawrence D'Oliveiro
Guest
Posts: n/a
 
      10-30-2010
In message <4cc5d9e9$0$1661$(E-Mail Removed)>, John Nagle wrote:

> Look at sock_close in "socketmodule.c". Note that it ignores the
> return status on close, always returns None, and never raises an
> exception. As the Linux manual page for "close" says:
> "Not checking the return value of close() is a common but nevertheless
> serious programming error. It is quite possible that errors on a
> previous write(2) operation are first reported at the final close(). Not
> checking the return value when closing the file may lead to silent loss
> of data."


The close call is the wrong place to report such errors. For output, there
should be some kind of flush-output call that you can use to get the error.
Close should just unconditionally tear down the connection, which you can do
whether the prior transfers were successful or not.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Get error message from FTPLib Bakes Python 1 09-23-2009 06:55 PM
How to get file name on a remote server with ftplib? alex Python 3 01-08-2007 10:15 AM
*WITHOUT* using: ValidateRequest="False" for the whole page (or my whole site).... \A_Michigan_User\ ASP .Net 2 08-21-2006 02:13 PM
ftplib - file.write, file.close() - verify finished writing locally before delete source? Peter A. Schott Python 0 02-10-2005 10:22 PM
Uploading a file with ftplib Michael_Goettsche Python 2 01-01-2004 02:56 PM



Advertisments