Panic Mode: A Cautionary Tale

Discussion in 'NZ Computing' started by Lawrence D'Oliveiro, Aug 3, 2007.

  1. Or, what happens when you're a rookie, and the boss is leaning on you to get
    the problem fixed, NOW.
    <http://weblog.infoworld.com/offtherecord/archives/2007/07/panic_mode.html>

    Oh yes, it also helps to ensure your backups are up-to-date.
     
    Lawrence D'Oliveiro, Aug 3, 2007
    #1
    1. Advertising

  2. Lawrence D'Oliveiro

    peterwn Guest

    Lawrence D'Oliveiro wrote:
    > Or, what happens when you're a rookie, and the boss is leaning on you to get
    > the problem fixed, NOW.
    > <http://weblog.infoworld.com/offtherecord/archives/2007/07/panic_mode.html>
    >
    > Oh yes, it also helps to ensure your backups are up-to-date.


    In the days just before PC's etc, we used dumb terminals off a
    mainframe. The mean sysadmin would not allow foreground compilations so
    you had to submit the compilation to background and come back a bit
    later. In particular the compiler listing was piped to a file for later
    examination.

    One guy came to me blind with panic - he had deleted his source file and
    no backup! Fortunately he still had the output file and I was able to
    'strip' the source code from the listing for him.
     
    peterwn, Aug 3, 2007
    #2
    1. Advertising

  3. Lawrence D'Oliveiro

    Matty F Guest

    On Aug 3, 9:01 pm, Lawrence D'Oliveiro <l...@geek-
    central.gen.new_zealand> wrote:
    > Or, what happens when you're a rookie, and the boss is leaning on you to get
    > the problem fixed, NOW.
    > <http://weblog.infoworld.com/offtherecord/archives/2007/07/panic_mode....>
    >
    > Oh yes, it also helps to ensure your backups are up-to-date.


    It also pays to try a restore occasionally, to see if your backups are
    actually working. In one large computer company, the log file was
    backed up every day. Unfortunately somebody changed the name of the
    file without telling the minion who backed it up. The minion backed up
    the old file for 6 months and filed the backup reports without
    noticing that the record counts and date last changed had stayed the
    same for 6 months.
     
    Matty F, Aug 4, 2007
    #3
  4. In message <>, Matty F
    wrote:

    > On Aug 3, 9:01 pm, Lawrence D'Oliveiro <l...@geek-
    > central.gen.new_zealand> wrote:
    >
    >> Oh yes, it also helps to ensure your backups are up-to-date.

    >
    > It also pays to try a restore occasionally, to see if your backups are
    > actually working.


    That's why my preferred form of backup is a simple rsync to another machine.
    No funny backup archive formats, just a straight file-by-file copy, easy to
    verify.
     
    Lawrence D'Oliveiro, Aug 5, 2007
    #4
  5. Lawrence D'Oliveiro

    Enkidu Guest

    Lawrence D'Oliveiro wrote:
    > In message <>, Matty F
    > wrote:
    >
    >> On Aug 3, 9:01 pm, Lawrence D'Oliveiro <l...@geek-
    >> central.gen.new_zealand> wrote:
    >>
    >>> Oh yes, it also helps to ensure your backups are up-to-date.

    >> It also pays to try a restore occasionally, to see if your backups are
    >> actually working.

    >
    > That's why my preferred form of backup is a simple rsync to another machine.
    > No funny backup archive formats, just a straight file-by-file copy, easy to
    > verify.
    >

    That's not very practical for Terabyte sized storage systems. OK for
    small users, I'd guess. Even just running through a TB sized file system
    and not copying anything can take hours.

    Cheers,

    Cliff

    --

    Have you ever noticed that if something is advertised as 'amusing' or
    'hilarious', it usually isn't?
     
    Enkidu, Aug 5, 2007
    #5
  6. Enkidu wrote:
    > Lawrence D'Oliveiro wrote:
    >> In message <>,
    >> Matty F
    >> wrote:
    >>
    >>> On Aug 3, 9:01 pm, Lawrence D'Oliveiro <l...@geek-
    >>> central.gen.new_zealand> wrote:
    >>>
    >>>> Oh yes, it also helps to ensure your backups are up-to-date.
    >>> It also pays to try a restore occasionally, to see if your backups are
    >>> actually working.

    >>
    >> That's why my preferred form of backup is a simple rsync to another
    >> machine.
    >> No funny backup archive formats, just a straight file-by-file copy,
    >> easy to
    >> verify.
    > >

    > That's not very practical for Terabyte sized storage systems. OK for
    > small users, I'd guess. Even just running through a TB sized file system
    > and not copying anything can take hours.
    >
    > Cheers,
    >
    > Cliff
    >

    2950 servers 4 TB disk pack configured in raid 5 takes just on 4 hours
    to robo copy changes
     
    collector«NZ, Aug 5, 2007
    #6
  7. Enkidu wrote:
    > Lawrence D'Oliveiro wrote:
    >> Matty F wrote:
    >>> On Aug 3, 9:01 pm, Lawrence D'Oliveiro wrote:
    >>>> Oh yes, it also helps to ensure your backups are up-to-date.
    >>> It also pays to try a restore occasionally, to see if your backups are
    >>> actually working.

    >> That's why my preferred form of backup is a simple rsync to another
    >> machine.
    >> No funny backup archive formats, just a straight file-by-file copy,
    >> easy to verify.

    >
    > That's not very practical for Terabyte sized storage systems. OK for
    > small users, I'd guess. Even just running through a TB sized file system
    > and not copying anything can take hours.


    What other backup technique is faster ?
     
    Mark Robinson, Aug 5, 2007
    #7
  8. In message <>, Enkidu wrote:

    > Lawrence D'Oliveiro wrote:
    >> In message <>, Matty
    >> F wrote:
    >>
    >>> On Aug 3, 9:01 pm, Lawrence D'Oliveiro <l...@geek-
    >>> central.gen.new_zealand> wrote:
    >>>
    >>>> Oh yes, it also helps to ensure your backups are up-to-date.
    >>> It also pays to try a restore occasionally, to see if your backups are
    >>> actually working.

    >>
    >> That's why my preferred form of backup is a simple rsync to another
    >> machine. No funny backup archive formats, just a straight file-by-file
    >> copy, easy to verify.
    >>

    > That's not very practical for Terabyte sized storage systems. OK for
    > small users, I'd guess. Even just running through a TB sized file system
    > and not copying anything can take hours.


    Depends on the number of files, rather than their size. rsync by default
    only checks file size and modification date before deciding whether it
    needs to do a full content comparison or not. (There is an option to
    override this.)
     
    Lawrence D'Oliveiro, Aug 5, 2007
    #8
  9. Lawrence D'Oliveiro

    Enkidu Guest

    collector«NZ wrote:
    > Enkidu wrote:
    >> Lawrence D'Oliveiro wrote:
    >>> In message <>,
    >>> Matty F
    >>> wrote:
    >>>
    >>>> On Aug 3, 9:01 pm, Lawrence D'Oliveiro <l...@geek-
    >>>> central.gen.new_zealand> wrote:
    >>>>
    >>>>> Oh yes, it also helps to ensure your backups are up-to-date.
    >>>> It also pays to try a restore occasionally, to see if your backups are
    >>>> actually working.
    >>>
    >>> That's why my preferred form of backup is a simple rsync to another
    >>> machine.
    >>> No funny backup archive formats, just a straight file-by-file copy,
    >>> easy to
    >>> verify.
    >> >

    >> That's not very practical for Terabyte sized storage systems. OK for
    >> small users, I'd guess. Even just running through a TB sized file
    >> system and not copying anything can take hours.
    >>
    >> Cheers,
    >>
    >> Cliff
    >>

    > 2950 servers 4 TB disk pack configured in raid 5 takes just on 4 hours
    > to robo copy changes
    >

    Yeah. It depends on file size and numbers. It also means that you need
    another 4TB somewhere. I copied 40 - 50 GB and it took about two hours.
    That was from one file system to another.

    Cheers,

    Cliff

    --

    Have you ever noticed that if something is advertised as 'amusing' or
    'hilarious', it usually isn't?
     
    Enkidu, Aug 5, 2007
    #9
  10. Lawrence D'Oliveiro

    Enkidu Guest

    Mark Robinson wrote:
    > Enkidu wrote:
    >> Lawrence D'Oliveiro wrote:
    >>> Matty F wrote:
    >>>> On Aug 3, 9:01 pm, Lawrence D'Oliveiro wrote:
    >>>>> Oh yes, it also helps to ensure your backups are up-to-date.
    >>>> It also pays to try a restore occasionally, to see if your backups are
    >>>> actually working.
    >>> That's why my preferred form of backup is a simple rsync to another
    >>> machine.
    >>> No funny backup archive formats, just a straight file-by-file copy,
    >>> easy to verify.

    >>
    >> That's not very practical for Terabyte sized storage systems. OK for
    >> small users, I'd guess. Even just running through a TB sized file
    >> system and not copying anything can take hours.

    >
    > What other backup technique is faster ?
    >

    If you find out, please let me know!

    Cheers,

    Cliff

    --

    Have you ever noticed that if something is advertised as 'amusing' or
    'hilarious', it usually isn't?
     
    Enkidu, Aug 5, 2007
    #10
  11. Lawrence D'Oliveiro

    Enkidu Guest

    Lawrence D'Oliveiro wrote:
    > In message <>, Enkidu wrote:
    >
    >> Lawrence D'Oliveiro wrote:
    >>> In message <>, Matty
    >>> F wrote:
    >>>
    >>>> On Aug 3, 9:01 pm, Lawrence D'Oliveiro <l...@geek-
    >>>> central.gen.new_zealand> wrote:
    >>>>
    >>>>> Oh yes, it also helps to ensure your backups are up-to-date.
    >>>> It also pays to try a restore occasionally, to see if your backups are
    >>>> actually working.
    >>> That's why my preferred form of backup is a simple rsync to another
    >>> machine. No funny backup archive formats, just a straight file-by-file
    >>> copy, easy to verify.
    >>>

    >> That's not very practical for Terabyte sized storage systems. OK for
    >> small users, I'd guess. Even just running through a TB sized file system
    >> and not copying anything can take hours.

    >
    > Depends on the number of files, rather than their size. rsync by default
    > only checks file size and modification date before deciding whether it
    > needs to do a full content comparison or not. (There is an option to
    > override this.)
    >

    Yes, but with file systems of Terabyte sizes even reading the file
    systems for file size and modification dates can take hours. Even
    without the file copy.

    Cheers,

    Cliff

    --

    Have you ever noticed that if something is advertised as 'amusing' or
    'hilarious', it usually isn't?
     
    Enkidu, Aug 5, 2007
    #11
  12. Enkidu wrote:
    > collector«NZ wrote:
    >> Enkidu wrote:
    >>> Lawrence D'Oliveiro wrote:
    >>>> In message <>,
    >>>> Matty F
    >>>> wrote:
    >>>>
    >>>>> On Aug 3, 9:01 pm, Lawrence D'Oliveiro <l...@geek-
    >>>>> central.gen.new_zealand> wrote:
    >>>>>
    >>>>>> Oh yes, it also helps to ensure your backups are up-to-date.
    >>>>> It also pays to try a restore occasionally, to see if your backups are
    >>>>> actually working.
    >>>>
    >>>> That's why my preferred form of backup is a simple rsync to another
    >>>> machine.
    >>>> No funny backup archive formats, just a straight file-by-file copy,
    >>>> easy to
    >>>> verify.
    >>> >
    >>> That's not very practical for Terabyte sized storage systems. OK for
    >>> small users, I'd guess. Even just running through a TB sized file
    >>> system and not copying anything can take hours.
    >>>
    >>> Cheers,
    >>>
    >>> Cliff
    >>>

    >> 2950 servers 4 TB disk pack configured in raid 5 takes just on 4 hours
    >> to robo copy changes
    > >

    > Yeah. It depends on file size and numbers. It also means that you need
    > another 4TB somewhere. I copied 40 - 50 GB and it took about two hours.
    > That was from one file system to another.
    >
    > Cheers,
    >
    > Cliff
    >

    My company is not disposed to spend much money, as a regional data
    centre I had to fight politics and bullshit to get diversity in my
    systems, so they got me the cheapest that matched my requirements and
    that was not a big expense at all
     
    collector«NZ, Aug 5, 2007
    #12
  13. Enkidu wrote:
    > Lawrence D'Oliveiro wrote:
    >> In message <>, Enkidu wrote:
    >>
    >>> Lawrence D'Oliveiro wrote:
    >>>> In message <>,
    >>>> Matty
    >>>> F wrote:
    >>>>
    >>>>> On Aug 3, 9:01 pm, Lawrence D'Oliveiro <l...@geek-
    >>>>> central.gen.new_zealand> wrote:
    >>>>>
    >>>>>> Oh yes, it also helps to ensure your backups are up-to-date.
    >>>>> It also pays to try a restore occasionally, to see if your backups are
    >>>>> actually working.
    >>>> That's why my preferred form of backup is a simple rsync to another
    >>>> machine. No funny backup archive formats, just a straight file-by-file
    >>>> copy, easy to verify.
    >>>>
    >>> That's not very practical for Terabyte sized storage systems. OK for
    >>> small users, I'd guess. Even just running through a TB sized file system
    >>> and not copying anything can take hours.

    >>
    >> Depends on the number of files, rather than their size. rsync by default
    >> only checks file size and modification date before deciding whether it
    >> needs to do a full content comparison or not. (There is an option to
    >> override this.)
    > >

    > Yes, but with file systems of Terabyte sizes even reading the file
    > systems for file size and modification dates can take hours. Even
    > without the file copy.
    >
    > Cheers,
    >
    > Cliff
    >

    As a mythical beast you might try examining the time involved, my 4TB
    (currently 2TB used) only takes 4 hours to mirror to an identical pack
     
    collector«NZ, Aug 5, 2007
    #13
  14. Lawrence D'Oliveiro

    peterwn Guest

    Matty F wrote:
    > On Aug 3, 9:01 pm, Lawrence D'Oliveiro <l...@geek-
    > central.gen.new_zealand> wrote:
    >> Or, what happens when you're a rookie, and the boss is leaning on you to get
    >> the problem fixed, NOW.
    >> <http://weblog.infoworld.com/offtherecord/archives/2007/07/panic_mode....>
    >>
    >> Oh yes, it also helps to ensure your backups are up-to-date.

    >
    > It also pays to try a restore occasionally, to see if your backups are
    > actually working.


    A senior Telecom guy had to look for alternative employment some years
    ago after the Palmerston North telephone exchange database failed to
    restore, the whole series of backups were duds. After several hours of
    a phoneless city, they found a 6 month old backup that restored which
    got most things going again except new numbers, relocations etc done in
    the previous 6 months.
     
    peterwn, Aug 5, 2007
    #14
  15. Lawrence D'Oliveiro

    Dave Doe Guest

    In article <>, says...
    > Enkidu wrote:
    > > Lawrence D'Oliveiro wrote:
    > >> In message <>,
    > >> Matty F
    > >> wrote:
    > >>
    > >>> On Aug 3, 9:01 pm, Lawrence D'Oliveiro <l...@geek-
    > >>> central.gen.new_zealand> wrote:
    > >>>
    > >>>> Oh yes, it also helps to ensure your backups are up-to-date.
    > >>> It also pays to try a restore occasionally, to see if your backups are
    > >>> actually working.
    > >>
    > >> That's why my preferred form of backup is a simple rsync to another
    > >> machine.
    > >> No funny backup archive formats, just a straight file-by-file copy,
    > >> easy to
    > >> verify.
    > > >

    > > That's not very practical for Terabyte sized storage systems. OK for
    > > small users, I'd guess. Even just running through a TB sized file system
    > > and not copying anything can take hours.
    > >
    > > Cheers,
    > >
    > > Cliff
    > >

    > 2950 servers 4 TB disk pack configured in raid 5 takes just on 4 hours
    > to robo copy changes


    This is great until the day your PSU catastrophically fails and takes
    out all your HDD's. (Happened to a server just a few months back - so I
    know it can happen).

    I appreciate it's a question of: how much is the data worth, and how
    much downtime can be afforded.

    I was lucky and able to get a drive working again by swapping controller
    cards from new drives and managed to get one working. They were all IDE
    drives in two RAID's on a budget server. Only the DVD was salvaged from
    the old server (ie the only working component left after the PSU
    failed).

    In short, if the data's important (and it usually is), offsite backups
    are essential.

    --
    Duncan
     
    Dave Doe, Aug 6, 2007
    #15
  16. In message <46b59eab$>, collector«NZ wrote:

    > As a mythical beast you might try examining the time involved, my 4TB
    > (currently 2TB used) only takes 4 hours to mirror to an identical pack


    And as another comparison, I have had one backup job that handles not quite
    200GB of data, but it consists of several million files. A daily run might
    take 20 minutes or so. rsync would actually run out of RAM (on a 1GiB
    machine) trying to build the directory tree, so I had to write a script to
    break up the backup by subdirectories--why couldn't rsync figure this out
    itself?
     
    Lawrence D'Oliveiro, Aug 6, 2007
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. MikeH
    Replies:
    4
    Views:
    1,829
    Karl \Johnno\ Gustaf
    Oct 29, 2003
  2. English Patient
    Replies:
    3
    Views:
    1,976
    Old Gringo
    Oct 4, 2004
  3. Replies:
    1
    Views:
    451
  4. RichA

    Adobe goes into back-peddling panic mode

    RichA, Jun 4, 2013, in forum: Digital Photography
    Replies:
    65
    Views:
    773
    Whisky-dave
    Jun 27, 2013
  5. Peter Huebner

    UEFI - a game of musical chairs: a cautionary tale.

    Peter Huebner, Dec 13, 2013, in forum: NZ Computing
    Replies:
    2
    Views:
    261
    Enkidu
    Dec 15, 2013
Loading...

Share This Page