How Do You Copy 60 Million Files From One Windows Server To Another?

Discussion in 'NZ Computing' started by Lawrence D'Oliveiro, Oct 3, 2010.

  1. Lawrence D'Oliveiro, Oct 3, 2010
    #1
    1. Advertisements

  2. Lawrence D'Oliveiro

    Ray Greene Guest

    Use Robocopy.

    From the article:
    XCopy and Robocopy most likely would have been able to handle the file
    volume but - like Windows Explorer - they are bound by the fact that
    NTFS can store files with longer names and greater path than CMD can
    handle

    From the 2003 Robocopy help file:
    The program has now been enhanced to handle pathnames of any
    length, up to the internal Windows limit of nearly 32,000 characters.

    He could have saved himself an awful lot of work if he had just
    updated a single ~200kb file sometime in the last seven years. Maybe
    he doesn't have to copy files very often.
     
    Ray Greene, Oct 4, 2010
    #2
    1. Advertisements

  3. Didn’t work for him:

    Robocopy theoretically /should/ see files with a path depth longer than
    255 characters, but in my experience simply doesn’t. It will see a file
    with a /name/ larger than 255 characters (or at least that’s what the
    length of the file name looks like at first glance,) however when I feed
    it /path depths/ larger than 255 is continually refuses to copy the
    file. I banged away at it for about half an hour before giving up and
    moving on to the next tool on my list. It should be noted that I tried
    only the command line version of robocopy. I did not give the GUI loader
    much of a go. (I figured if I was going to faff about with GUI tools,
    Richcopy > Robocopy anyways, so….)

    Also, it still would have been constrained by Windows’ performance
    limitations.

    In one of his followup comments, Trevor said the best a single-threaded
    Windows-based copy tool could do was 38 hours for the whole copy, versus 31
    hours for Linux doing it. (He didn’t want to use a multi-threaded copy
    because of the usual Windows fragmentation problems.)
     
    Lawrence D'Oliveiro, Oct 4, 2010
    #3
  4. Lawrence D'Oliveiro

    Ray Greene Guest

    I've used Robocopy on files with horrendously long paths and never had
    a problem. It would be interesting to know how long the paths on his
    server were, and what version he was actually using.
    That's a considerable difference. I'd like to speed test cp against
    Robocopy some time to see how they compare.
     
    Ray Greene, Oct 5, 2010
    #4
  5. Lawrence D'Oliveiro

    Richard Guest

    Ive used robocopy for the same thing in the past to move things into my
    music collection when the long album and track names have pushed it over
    the length that windows explorer will let you do.

    Same for when idiots have made torrents with spaces at the start of the
    name, that explorer helpfully removes for you with no say in the matter.
     
    Richard, Oct 5, 2010
    #5
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.