How Do You Copy 60 Million Files From One Windows Server To Another?

Discussion in 'NZ Computing' started by Lawrence D'Oliveiro, Oct 3, 2010.

  1. Use Linux.

    <http://www.theregister.co.uk/2010/09/24/sysadmin_file_tools/>
     
    Lawrence D'Oliveiro, Oct 3, 2010
    #1
    1. Advertising

  2. Lawrence D'Oliveiro

    Ray Greene Guest

    On Sun, 03 Oct 2010 23:24:09 +1300, Lawrence D'Oliveiro
    <_zealand> wrote:

    >Use Linux.


    Use Robocopy.

    From the article:
    XCopy and Robocopy most likely would have been able to handle the file
    volume but - like Windows Explorer - they are bound by the fact that
    NTFS can store files with longer names and greater path than CMD can
    handle

    From the 2003 Robocopy help file:
    The program has now been enhanced to handle pathnames of any
    length, up to the internal Windows limit of nearly 32,000 characters.

    He could have saved himself an awful lot of work if he had just
    updated a single ~200kb file sometime in the last seven years. Maybe
    he doesn't have to copy files very often.

    --
    Ray Greene
     
    Ray Greene, Oct 4, 2010
    #2
    1. Advertising

  3. In message <>, Ray Greene wrote:

    > On Sun, 03 Oct 2010 23:24:09 +1300, Lawrence D'Oliveiro
    > <_zealand> wrote:
    >
    >>Use Linux.

    >
    > Use Robocopy.


    Didn’t work for him:

    Robocopy theoretically /should/ see files with a path depth longer than
    255 characters, but in my experience simply doesn’t. It will see a file
    with a /name/ larger than 255 characters (or at least that’s what the
    length of the file name looks like at first glance,) however when I feed
    it /path depths/ larger than 255 is continually refuses to copy the
    file. I banged away at it for about half an hour before giving up and
    moving on to the next tool on my list. It should be noted that I tried
    only the command line version of robocopy. I did not give the GUI loader
    much of a go. (I figured if I was going to faff about with GUI tools,
    Richcopy > Robocopy anyways, so….)

    Also, it still would have been constrained by Windows’ performance
    limitations.

    In one of his followup comments, Trevor said the best a single-threaded
    Windows-based copy tool could do was 38 hours for the whole copy, versus 31
    hours for Linux doing it. (He didn’t want to use a multi-threaded copy
    because of the usual Windows fragmentation problems.)
     
    Lawrence D'Oliveiro, Oct 4, 2010
    #3
  4. Lawrence D'Oliveiro

    Ray Greene Guest

    On Tue, 05 Oct 2010 09:59:06 +1300, Lawrence D'Oliveiro
    <_zealand> wrote:

    >In message <>, Ray Greene wrote:
    >
    >> On Sun, 03 Oct 2010 23:24:09 +1300, Lawrence D'Oliveiro
    >> <_zealand> wrote:
    >>
    >>>Use Linux.

    >>
    >> Use Robocopy.

    >
    >Didn’t work for him:
    >
    > Robocopy theoretically /should/ see files with a path depth longer than
    > 255 characters, but in my experience simply doesn’t. It will see a file
    > with a /name/ larger than 255 characters (or at least that’s what the
    > length of the file name looks like at first glance,) however when I feed
    > it /path depths/ larger than 255 is continually refuses to copy the
    > file. I banged away at it for about half an hour before giving up and
    > moving on to the next tool on my list. It should be noted that I tried
    > only the command line version of robocopy. I did not give the GUI loader
    > much of a go. (I figured if I was going to faff about with GUI tools,
    > Richcopy > Robocopy anyways, so….)


    I've used Robocopy on files with horrendously long paths and never had
    a problem. It would be interesting to know how long the paths on his
    server were, and what version he was actually using.

    >Also, it still would have been constrained by Windows’ performance
    >limitations.
    >
    >In one of his followup comments, Trevor said the best a single-threaded
    >Windows-based copy tool could do was 38 hours for the whole copy, versus 31
    >hours for Linux doing it. (He didn’t want to use a multi-threaded copy
    >because of the usual Windows fragmentation problems.)


    That's a considerable difference. I'd like to speed test cp against
    Robocopy some time to see how they compare.

    --
    Ray Greene
     
    Ray Greene, Oct 5, 2010
    #4
  5. Lawrence D'Oliveiro

    Richard Guest

    On 5/10/2010 8:05 p.m., Ray Greene wrote:
    > On Tue, 05 Oct 2010 09:59:06 +1300, Lawrence D'Oliveiro
    > <_zealand> wrote:
    >
    >> In message<>, Ray Greene wrote:
    >>
    >>> On Sun, 03 Oct 2010 23:24:09 +1300, Lawrence D'Oliveiro
    >>> <_zealand> wrote:
    >>>
    >>>> Use Linux.
    >>>
    >>> Use Robocopy.

    >>
    >> Didn’t work for him:
    >>
    >> Robocopy theoretically /should/ see files with a path depth longer than
    >> 255 characters, but in my experience simply doesn’t. It will see a file
    >> with a /name/ larger than 255 characters (or at least that’s what the
    >> length of the file name looks like at first glance,) however when I feed
    >> it /path depths/ larger than 255 is continually refuses to copy the
    >> file. I banged away at it for about half an hour before giving up and
    >> moving on to the next tool on my list. It should be noted that I tried
    >> only the command line version of robocopy. I did not give the GUI loader
    >> much of a go. (I figured if I was going to faff about with GUI tools,
    >> Richcopy> Robocopy anyways, so….)

    >
    > I've used Robocopy on files with horrendously long paths and never had
    > a problem. It would be interesting to know how long the paths on his
    > server were, and what version he was actually using.
    >
    >> Also, it still would have been constrained by Windows’ performance
    >> limitations.
    >>
    >> In one of his followup comments, Trevor said the best a single-threaded
    >> Windows-based copy tool could do was 38 hours for the whole copy, versus 31
    >> hours for Linux doing it. (He didn’t want to use a multi-threaded copy
    >> because of the usual Windows fragmentation problems.)

    >
    > That's a considerable difference. I'd like to speed test cp against
    > Robocopy some time to see how they compare.
    >


    Ive used robocopy for the same thing in the past to move things into my
    music collection when the long album and track names have pushed it over
    the length that windows explorer will let you do.

    Same for when idiots have made torrents with spaces at the start of the
    name, that explorer helpfully removes for you with no say in the matter.
     
    Richard, Oct 5, 2010
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Terry

    Transfering files from one computer to another one

    Terry, Jul 14, 2003, in forum: Microsoft Certification
    Replies:
    5
    Views:
    9,224
    Chris Kirk
    Jul 25, 2003
  2. James
    Replies:
    2
    Views:
    575
    Walter Roberson
    Jul 21, 2003
  3. Film Buff

    Fox Screws up "One Million B.C."

    Film Buff, Mar 2, 2004, in forum: DVD Video
    Replies:
    1
    Views:
    573
    Thomas Bell
    Mar 2, 2004
  4. Guyster
    Replies:
    1
    Views:
    366
    Walter Roberson
    Nov 22, 2006
  5. Missing Madeleine McCann

    A L E R T MISSING CHILD AGE 4 CAN YOU HELP? REWARD 4 MILLION EUROS

    Missing Madeleine McCann, May 13, 2007, in forum: Computer Support
    Replies:
    4
    Views:
    2,640
Loading...

Share This Page