Disaster Recovery Site

I hope that this is an acceptable topic for this ng. I would
appreciate knowing if there is another one that is more
appropriate; I couldn't find one.

We are a not-for-profit, primarily involved in clinical trials.
We have reasonably good facilities to deal with emergiencies
(diesel generator, backup cooling, independent internet
links, etc), we do not have a disaster recovery site. The
reality is that we have only one application that is highly
sensitive to downtime, and that one can be performed
manually. We have the procedures to do switch over to
that at any time. But if we were to be out for more than
three days, the problems would begin to build up. And
we do have a large number of applications that are in
many cases unique to individual clients, and many of
these should be made available if we were to face a
long down time of our primary site.

We started work to establish a disaster recovery site
at a sister office approx. 1,000 miles away. After considering
all of the costs, we have come around to considering the
possibility of establishing the site in a building in our office
complex several hundred yards away from ours and to which
we plan to expand in the next year. There would be a number
of advantages to setting up the disaster site in that building
from our point of view, both in terms of functionality and cost.

We have tried to ennumerate and assign a probability to
all of the possible events we could imagine that would bring
down the primary site and the secondary site at the same
time. Our conclusion was that the chances of the latter were
extremely small, with one exception: an area-wide communi-
cations failure. We thought of several possible solutions,
but the most cost-effective would be backup sattelite service.
This of course would be a low-bandwith solution but it appears
to us to be a functionally exceptable one, even if we do not
put users on rationed access times. If both sites were in
fact destroyed we would have very serious problems until
recovery, but we would have offsite backups that would be
up to date, so ultimately we would be able to recover data-
bases to a point very close to the failure point.

The questions that we have are 1) is this a practice followed
by any other organizations, and 2), do people believe that it
is defensible on the grounds I layed out above ? Any input
you would give me will be very much appreciated by me.

Thank you !

pavlov

On Sat, 26 Mar 2005 13:08:22 -0500, pavlov <>
wrote:

>This of course would be a low-bandwith solution but it appears
>to us to be a functionally exceptable one, even if we do not
>put users on rationed access times.


Sorry, I royally mucked up some sentences, especially the
above: "acceptable," not "exceptable."

pavlov

Did you look into services such as this?:

http://www.iprevolution.com/business_continuity.html

pavlov wrote:

> I hope that this is an acceptable topic for this ng. I would
> appreciate knowing if there is another one that is more
> appropriate; I couldn't find one.
>
> We are a not-for-profit, primarily involved in clinical trials.
> We have reasonably good facilities to deal with emergiencies
> (diesel generator, backup cooling, independent internet
> links, etc), we do not have a disaster recovery site. The
> reality is that we have only one application that is highly
> sensitive to downtime, and that one can be performed
> manually. We have the procedures to do switch over to
> that at any time. But if we were to be out for more than
> three days, the problems would begin to build up. And
> we do have a large number of applications that are in
> many cases unique to individual clients, and many of
> these should be made available if we were to face a
> long down time of our primary site.
>
> We started work to establish a disaster recovery site
> at a sister office approx. 1,000 miles away. After considering
> all of the costs, we have come around to considering the
> possibility of establishing the site in a building in our office
> complex several hundred yards away from ours and to which
> we plan to expand in the next year. There would be a number
> of advantages to setting up the disaster site in that building
> from our point of view, both in terms of functionality and cost.
>
> We have tried to ennumerate and assign a probability to
> all of the possible events we could imagine that would bring
> down the primary site and the secondary site at the same
> time. Our conclusion was that the chances of the latter were
> extremely small, with one exception: an area-wide communi-
> cations failure. We thought of several possible solutions,
> but the most cost-effective would be backup sattelite service.
> This of course would be a low-bandwith solution but it appears
> to us to be a functionally exceptable one, even if we do not
> put users on rationed access times. If both sites were in
> fact destroyed we would have very serious problems until
> recovery, but we would have offsite backups that would be
> up to date, so ultimately we would be able to recover data-
> bases to a point very close to the failure point.
>
> The questions that we have are 1) is this a practice followed
> by any other organizations, and 2), do people believe that it
> is defensible on the grounds I layed out above ? Any input
> you would give me will be very much appreciated by me.
>
> Thank you !

Ralph A. Jones

pavlov wrote:
> I hope that this is an acceptable topic for this ng. I would
> appreciate knowing if there is another one that is more
> appropriate; I couldn't find one.

spot on

and you expressed your problem clearly as well, but I snipped it for brevity

I am sure it will raise a lot of debate, so here is my take....and you
get what you pay for

Many of my clients are not-for-profits/charities
Most of them didn't even have backups
I come from a background of consultancy and network security before
doing my own thing

so, from there...

You seem to be doing the right things, have a brainstorming session and
get all the risks and consequences down on paper. I'd call this a
FMEACA, failure mode, effects, and consequences analysis. Make sure you
get (very) senior manager buy in at this stage.

Then put a probability on each failure, and a nominal cost. No more than
five categories for each, say; likely, maybe, unlikely; expensive,
moderate, cheap with a final column relating risk & consequence - high,
medium, low (say) or just a nominal cost by multiplying risk and
consequence factors

draw up a table

put in costs to mitigate each failure mode and a VERY brief outline of a
plan (not more than 5 bullet points)

get it agreed by your manaegement team (it's their company)

Then get everyone to have a meeting and thrash out how much cash they
want to spend on the total plan. Then allocate your spending (on a 5
year budget) to go from highest to lowest total risk.

If anything is missed, you have a committee to sack not just you You
can't do this on your own, you need your company to back you to the hilt
on it and get their buy-in

Oh, it's hard to get non-profits/charities to actually spend money on
this kind of stuff, even getting backups made is often painful

<cut>

Martin

On Sat, 26 Mar 2005 15:52:26 -0600, "Ralph A. Jones"
<rajones@SPAM_ME_NOT_AT_tconl.com> wrote:

>Did you look into services such as this?:
>
>http://www.iprevolution.com/business_continuity.html

Yes, and that solution is still a possibility: we have pricing
from several vendors for various configurations and
capacities. But we believe that the homegrown solution
can be more cost-effective for us, especially if we can
accomplish a few things that we are exploring now.

Thank you for the input.

pavlov

On Sat, 26 Mar 2005 22:46:26 +0000 (UTC), Martin
<> wrote:


>I am sure it will raise a lot of debate, so here is my take....and you
>get what you pay for
>

It sounds fine to me. I don't really have to worry very much about
senior management, but if I'm to look beyond the functional utility of
what you propose to the political utility, it appears that if we
follow through as you suggest, we should be on solid ground if an
auditor of some sort should pop in. Or if we have a single event that
destroys both sites

Thanks for the thoughtful response.

pavlov

On Sat, 26 Mar 2005 18:16:18 -0500, pavlov wrote:
>
> On Sat, 26 Mar 2005 15:52:26 -0600, "Ralph A. Jones"
> <rajones@SPAM_ME_NOT_AT_tconl.com> wrote:
>
>>Did you look into services such as this?:
>>
>>http://www.iprevolution.com/business_continuity.html
>
> Yes, and that solution is still a possibility: we have pricing
> from several vendors for various configurations and
> capacities. But we believe that the homegrown solution
> can be more cost-effective for us, especially if we can
> accomplish a few things that we are exploring now.
>
> Thank you for the input.

Why not just setup a fiber connection between buildings, add a couple
servers to the domain, put them in the second building, replicate files on
a nightly (or quicker if needed) basis between servers, and be happy.

--

remove 999 in order to email me

Leythos

On Sun, 27 Mar 2005 00:58:26 GMT, Leythos <> wrote:

>
>Why not just setup a fiber connection between buildings, add a couple
>servers to the domain, put them in the second building, replicate files on
>a nightly (or quicker if needed) basis between servers, and be happy.

That is a simplified version of what we have in mind. My
concern is whether the lack of any real geographic
separation will be held against us.

pavlov

On Sat, 26 Mar 2005 21:16:56 -0500, pavlov wrote:
>
> On Sun, 27 Mar 2005 00:58:26 GMT, Leythos <> wrote:
>
>
>>Why not just setup a fiber connection between buildings, add a couple
>>servers to the domain, put them in the second building, replicate files
>>on a nightly (or quicker if needed) basis between servers, and be happy.
>
> That is a simplified version of what we have in mind. My concern is
> whether the lack of any real geographic separation will be held against
> us.

If the building is in the same earth-quake area, close enough for a fire
to spread around the compound, EMP, Power loss in a grid, Flooding, etc...
If you can't afford to be down for XX hours, then you need to move the
your backup center to another location.

As a side note, we designed a medical center with remote offices, all data
is central to the main office compound. Backups are several layers, but
final backup is to tape - we contract with an off-site storage place that
sends a person to pick up tapes and return the prior ones each day. Some
locations don't do any off-site backup, just have a UPS setup that's
capable of 24 hours service, but their remote offices backup to the main
office nightly.

If your process/data is critical enough that you can't be down for 1 day
in the event of a complete disaster, then you want the remote recovery
location to be in another city where none of the items above can impact
both sites at the same time from the same event.

--

remove 999 in order to email me

Leythos