Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > copying files

Reply
Thread Tools

copying files

 
 
Hans Vlems
Guest
Posts: n/a
 
      02-16-2012
On 16 feb, 19:11, Malcolm McLean <(E-Mail Removed)>
wrote:
> On Feb 16, 5:56*pm, Keith Thompson <(E-Mail Removed)> wrote:
>
> > Don't assume that copying a byte at a time will be slow. *stdio does its
> > own buffering internally; most calls to getc will just read a byte from
> > a memory buffer.

>
> Yes, there's not much point implementing a buffer on top of another
> buffer. You are doing the test for EOF on every loop iteration, but
> that won't make much of a difference.


Malcolm,
I tried to improve on performance and in an earlier reply I wrote that
the original code you provided was sufficiently fast for the files I
fed it. So it doesn't matter much for my purpose. I'm dealing with
fairly small files here: they're all below 10 MB. What I did want is
to know whether something went wrong or not during a copy operation.
Which is why I can't use system(). It signals the status of passing a
command (script) to the OS. I'd rather stay away from programming DOS
command scripts and error handling therein! IMHO that's why
programming languages were invented... So don't worry, your code waas
very helpful. It's the goto's that had to go...
Hans
 
Reply With Quote
 
 
 
 
Keith Thompson
Guest
Posts: n/a
 
      02-16-2012
Hans Vlems <(E-Mail Removed)> writes:
> On 16 feb, 19:11, Malcolm McLean <(E-Mail Removed)>
> wrote:
>> On Feb 16, 5:56*pm, Keith Thompson <(E-Mail Removed)> wrote:
>>
>> > Don't assume that copying a byte at a time will be slow. *stdio does its
>> > own buffering internally; most calls to getc will just read a byte from
>> > a memory buffer.

>>
>> Yes, there's not much point implementing a buffer on top of another
>> buffer. You are doing the test for EOF on every loop iteration, but
>> that won't make much of a difference.

>
> I tried to improve on performance and in an earlier reply I wrote that
> the original code you provided was sufficiently fast for the files I
> fed it. So it doesn't matter much for my purpose. I'm dealing with
> fairly small files here: they're all below 10 MB. What I did want is
> to know whether something went wrong or not during a copy operation.
> Which is why I can't use system(). It signals the status of passing a
> command (script) to the OS. I'd rather stay away from programming DOS
> command scripts and error handling therein! IMHO that's why
> programming languages were invented... So don't worry, your code waas
> very helpful. It's the goto's that had to go...


Calling fclose() and having it return zero to indicate success
doesn't actually guarantee that the bits ended up on the platter.
The C standards says "Any unwritten buffered data for the stream
are delivered to the host environment to be written to the file".

Whatever method you use to copy the files, the only way to be 100%
sure that the file was actually copied is to read the copy and
compare it to the original.

--
Keith Thompson (The_Other_Keith) http://www.velocityreviews.com/forums/(E-Mail Removed) <http://www.ghoti.net/~kst>
Will write code for food.
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"
 
Reply With Quote
 
 
 
 
Nobody
Guest
Posts: n/a
 
      02-17-2012
On Thu, 16 Feb 2012 09:41:42 -0800, Keith Thompson wrote:

>> Avoid system() unless executing a "canned" command supplied by the user.
>> If you need to spawn a child process with specific arguments, use fork()
>> and exec*() rather than attempting to construct a shell command.

> [...]
>
> This, as well as the rest of your response, relies heavily on the
> assumption that the OP is on a Unix-like system.


True, but "equivalent" issues exist on other platforms. If anything,
Windows is even worse (both on the system() issue and on filesystem
semantics).

 
Reply With Quote
 
Malcolm McLean
Guest
Posts: n/a
 
      02-17-2012
On Feb 16, 11:11*pm, Hans Vlems <(E-Mail Removed)> wrote:
> On 16 feb, 19:11, Malcolm McLean <(E-Mail Removed)>
> What I did want is to know whether something went wrong or not during a copy operation.
>

fopen return null if it fails.
fgetc() returns EOF on end of file, or error.
So after fgetc() returns EOF, call feof() to see whether the EOF was
generated by a read error or by end of file.
fputc() returns EOF of fail.
fclose() returns EOF on failure. Most people don't bother to check
this. But it's possible to have write error as the last few bytes are
flushed out.

As Keith says, the only way to be absolutely sure the file is
physically written is to read it back in. Even then, it could become
corrupted on disk. But that sort of error is very rare. Almost always
the error will be because fopen() fails, either it has been passed a
bad filename or the device isn't working properly.
--
Vist my website
http://www.malcolmmclean.site11.com/www

 
Reply With Quote
 
Philip Lantz
Guest
Posts: n/a
 
      02-17-2012
Keith Thompson wrote:
> Calling fclose() and having it return zero to indicate success
> doesn't actually guarantee that the bits ended up on the platter.
> The C standards says "Any unwritten buffered data for the stream
> are delivered to the host environment to be written to the file".
>
> Whatever method you use to copy the files, the only way to be 100%
> sure that the file was actually copied is to read the copy and
> compare it to the original.


Reading the copy and comparing it to the original also doesn't actually
guarantee that the bits ended up on the platter.
 
Reply With Quote
 
Willem
Guest
Posts: n/a
 
      02-17-2012
Malcolm McLean wrote:
) As Keith says, the only way to be absolutely sure the file is
) physically written is to read it back in. Even then, it could become
) corrupted on disk.

Or still be hanging in the volatile write cache, still scheduled to be
actually written. Which is more more likely. And if you're writing to
a network share, all bets are off.

If you really really want to be sure a file is actually on the disk
platters, you need OS-specific support.


SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT
 
Reply With Quote
 
Hans Vlems
Guest
Posts: n/a
 
      02-18-2012
On 17 feb, 12:36, Malcolm McLean <(E-Mail Removed)>
wrote:
> On Feb 16, 11:11*pm, Hans Vlems <(E-Mail Removed)> wrote:> On 16 feb, 19:11, Malcolm McLean <(E-Mail Removed)>
> > What I did want is to know whether something went wrong or not during acopy operation.

>
> fopen return null if it fails.
> fgetc() returns EOF on end of file, or error.
> So after fgetc() returns EOF, call feof() to see whether the EOF was
> generated by a read error or by end of file.
> fputc() returns EOF of fail.
> fclose() returns EOF on failure. Most people don't bother to check
> this. But it's possible to have write error as the last few bytes are
> flushed out.
>
> As Keith says, the only way to be absolutely sure the file is
> physically written is to read it back in. Even then, it could become
> corrupted on disk. But that sort of error is very rare. Almost always
> the error will be because fopen() fails, either it has been passed a
> bad filename or the device isn't working properly.
> --
> Vist my websitehttp://www.malcolmmclean.site11.com/www


The error handling available for fputc() and fgetc() is indeed very
useful.
COPY&COMPARE is a valid strategy if you're running on unreliable
disks.
In the mean time I got more information from the ICT support people.
Their view
is that the disks themselves are in good shape, but the configuration
of the SAN
and the Citrix farm connected to it were both poorly designed.
They're working to improve the setup but money is tight. That said,
comparing the
two copies might be the best strategy for the moment.
Hans
 
Reply With Quote
 
Hans Vlems
Guest
Posts: n/a
 
      02-18-2012
On 17 feb, 18:53, Philip Lantz <(E-Mail Removed)> wrote:
> Keith Thompson wrote:
> > Calling fclose() and having it return zero to indicate success
> > doesn't actually guarantee that the bits ended up on the platter.
> > The C standards says "Any unwritten buffered data for the stream
> > are delivered to the host environment to be written to the file".

>
> > Whatever method you use to copy the files, the only way to be 100%
> > sure that the file was actually copied is to read the copy and
> > compare it to the original.

>
> Reading the copy and comparing it to the original also doesn't actually
> guarantee that the bits ended up on the platter.


Correct. As others already mentioned here, there are multiple layares
of caches in the
path between data in memory and data on disk.
 
Reply With Quote
 
Hans Vlems
Guest
Posts: n/a
 
      02-18-2012
On 17 feb, 23:04, Willem <(E-Mail Removed)> wrote:
> Malcolm McLean wrote:
>
> ) As Keith says, the only way to be absolutely sure the file is
> ) physically written is to read it back in. Even then, it could become
> ) corrupted on disk.
>
> Or still be hanging in the volatile write cache, still scheduled to be
> actually written. *Which is more more likely. *And if you're writing to
> a network share, all bets are off.
>
> If you really really want to be sure a file is actually on the disk
> platters, you need OS-specific support.
>
> SaSW, Willem
> --
> Disclaimer: I am in no way responsible for any of the statements
> * * * * * * made in the above text. For all I know I might be
> * * * * * * drugged or something..
> * * * * * * No I'm not paranoid. You all think I'm paranoid, don't you !
> #EOT


OTOH Willem if you cannot rely on the firmware to put cached data
correctly on disk
then you're definitely in need of a better disk subsystem!
I'm not sure I understand your reference to "OS-specific
support"though.
Do you mean the existence of a record management system (e.g. RMS, a
VMS component)?
Windows and unix(es) don't have this feature and the programmer is
solely resposible
for the desired file structure.
A record management system may be called to report on errors etc. Was
that your point?
Hans
 
Reply With Quote
 
Willem
Guest
Posts: n/a
 
      02-18-2012
Hans Vlems wrote:
) On 17 feb, 23:04, Willem <(E-Mail Removed)> wrote:
)> Malcolm McLean wrote:
)>
)> ) As Keith says, the only way to be absolutely sure the file is
)> ) physically written is to read it back in. Even then, it could become
)> ) corrupted on disk.
)>
)> Or still be hanging in the volatile write cache, still scheduled to be
)> actually written. ?Which is more more likely. ?And if you're writing to
)> a network share, all bets are off.
)>
)> If you really really want to be sure a file is actually on the disk
)> platters, you need OS-specific support.
)
) OTOH Willem if you cannot rely on the firmware to put cached data
) correctly on disk
) then you're definitely in need of a better disk subsystem!

I wasn't talking about the firmware layer, I was talking about
the OS write caching layer.

) I'm not sure I understand your reference to "OS-specific
) support"though.

For example, the sync() and fsync() call in several unix flavours.


SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Copying files to Vista Program Files in ruby script Shawn Mcclain Ruby 0 09-28-2007 09:21 PM
XCOPY is not copying ascx files while it copies .dll files c.verma@gmail.com ASP .Net 0 01-14-2005 07:29 PM
copying multiple files yaduraj Perl 1 08-09-2004 06:06 PM
Copying files ......... ALPO ASP .Net 1 12-20-2003 09:33 PM
moving and copying encrypted files Paul L MCSE 8 11-04-2003 03:21 AM



Advertisments