Disaster Recovery That Works When You Need It

How much recent work can this system afford to lose, and how long can it be unavailable. Those two answers drive every technical choice that follows, and both of them belong to the business rather than to whoever runs the servers.

The first is the recovery point. If your last usable copy is from midnight and the failure happens at four in the afternoon, everything done since midnight has to be recreated by somebody, from memory, from paper, or from emails. For a document store that might be irritating. For a system taking orders it can be worse than the outage itself.

The second is the recovery time, measured from the moment the decision is made rather than from the moment the engineers start typing. It includes noticing, escalating, deciding, and all the arguing that happens before anybody touches a keyboard. Plans that measure only the technical work produce numbers that are wrong by hours.

Ask the owner of each significant system both questions in ordinary language. When does this stop being annoying and start being serious. And if we brought it back as it was this morning, what would somebody have to redo by hand. The answers differ far more between systems than most people expect, and one policy for everything means overpaying for the systems that do not need it and underprotecting the ones that do.

Backup answers a question about data: somebody deleted a folder, a table got corrupted, a file needs to be as it was last Tuesday. Disaster recovery answers a question about service: the place where this runs is gone, and the business needs to work again.

The distinction matters because backups alone give you a recovery time nobody has measured. Having the data is the first step of a sequence that also includes building the environment, restoring into it, starting the applications in the right order, pointing the network at the new place, and confirming with somebody who uses the system that it is actually working.

That sequence is where the hours go. A restore that copies data quickly can still leave you a day away from a working business, because the environment it needs to land in does not exist yet and nobody has practised building it under pressure.

A backup that has never been restored is a hypothesis with a good filing system. The job finished, the report is green, the storage is being paid for. None of that is evidence that the data is usable, that the application will start, or that anybody knows the procedure. The only evidence is a restore somebody has actually done.

The cheapest is a cold start. You hold backups and the definitions needed to build the environment, and you build it when something goes wrong. Running cost is close to nothing beyond storage. Recovery is measured in hours or days, depending entirely on how much of the build is scripted and how much lives in somebody's head.

Next is a small standby. A minimal version of the environment exists all the time in another region: the network, the identity setup, the database receiving a copy of the data, and machines that are either off or very small. When something fails you grow it and switch over. Modest running cost, recovery in a manageable number of hours.

Then a warm standby, which is the same thing built at a size that could carry real traffic with some scaling. Costs are meaningful because you are running the important parts twice. Recovery is fast enough for systems where a morning offline would be a serious event.

And finally running in more than one place at once, with traffic served from both. It is the fastest to recover and by a distance the most expensive and most complex, because your application has to be built for it. Very few businesses need it for more than one or two systems, and plenty of businesses buy the complexity without the requirement.

The data usually arrives. What breaks is everything around it, and the list is remarkably consistent across the failovers we have watched.

Name records with a long cache life, so users keep being sent to a place that no longer answers, long after the switch was made. Third parties that only accept traffic from your original addresses, which means your bank, your payment processor and your logistics provider all stop talking to you at the same moment. Licence servers tied to hardware identifiers that changed. Certificates that were only ever installed in one place. Scheduled jobs that now exist in both environments and run twice, which for anything that sends invoices or moves stock is worse than not running at all.

Then the credentials. Secrets and keys stored in the region that is unavailable, needed by the environment that is supposed to take over. It is a circular dependency that reads as obvious on paper and is discovered during real events with tedious regularity.

And capacity, which is the one nobody wants to think about. A regional failure means every customer of that provider is trying to start machines in the neighbouring region at the same time. Capacity is not guaranteed unless you have reserved it. If your plan assumes you can create a large fleet on demand in a crisis, that is an assumption worth testing with your provider before you rely on it.

A virtual machine in one availability zone is one machine. If the hardware under it fails, it stops, exactly as a server in your own building would. The platform will usually restart it somewhere else, and that is a restart rather than continuity.

Managed services vary and the details are in the documentation rather than in the brochure. Some replicate across zones automatically. Some do it only if you chose the option and paid for it. Some are regional and will go with the region. Read the specific commitment for the specific service you are using, in the tier you bought, because the difference between a regional service and a zonal one is the difference between an incident and a disaster.

Object storage is generally the most resilient thing you own and the easiest to over trust. Durable storage protects you from hardware failure. It does not protect you from somebody deleting the bucket, from a mistake in your own code, or from an attacker with your credentials.

Being honest about this is uncomfortable in a sales conversation and cheap in an outage. Ask directly what happens if a whole region goes away, and if the answer is that the system stops, decide whether that is acceptable and write down the decision.

A plan tested only in a meeting is a plan with unknown recovery times. The test that produces real numbers is a failover run on an ordinary working day, with the business informed, during hours when the people involved are awake and the suppliers you might need are answering their phones.

Run the whole thing. Declare the event, follow the plan as written, bring the service up in the other place, and then let the business actually use it for the rest of the day. Half the defects in a recovery plan only appear when real users do real work: the integration nobody remembered, the report that points at the old address, the printer queue that does not exist in the new environment.

Then fail back, and treat that as part of the test. Failing back is often harder than failing over, because data has been created in the standby that now has to travel in the opposite direction, and very few plans describe that step in any detail.

Vary who runs it. If the recovery only works when one particular engineer is available, you have a dependency on somebody who may be on a plane. Run at least one test with that person unavailable and see how far the documentation carries the rest of the team.

Somebody has to say the word. In most badly handled events the technical recovery was fine and the delay was in the hour or two where everybody was waiting for a decision nobody had been given the authority to make.

Name the people who can declare, make that list longer and more junior than feels comfortable, and give them a threshold in concrete terms rather than in adjectives. If the order system has been unreachable for a stated length of time and the cause is not understood, declare. A recovery declared and stood down costs a little disruption. A recovery declared late costs the whole day.

The runbook has to live somewhere that survives the event. A procedure stored in a wiki behind the identity system that is down is not available at the one moment it is needed. Keep a copy outside the affected environment, carrying the steps, the order systems must come back in, the contacts, the account numbers and the support entitlements.

Include the going back decision as a written step with a time on it. Without that, teams will keep trying to fix the original environment for hours because switching over feels drastic and repairing feels close to done.

Very few businesses need the same protection for everything they run. Sort your systems into a small number of tiers by what an outage actually costs, and accept genuinely different answers. The systems that take money and get product out of the door earn a standby. The internal wiki does not.

Price each tier honestly, including the work to build it and the work to test it every year. A warm standby that nobody has budget to exercise will decay quietly until it is a line item rather than a capability.

It is worth saying plainly that for a lot of small and mid sized businesses, well tested backups with a scripted environment rebuild is a legitimate answer for most systems. It is not the impressive option and it recovers slower, and if the business can genuinely absorb that, the money is better spent elsewhere.

What is never acceptable is a documented recovery time that nobody has measured. Either test it and write down the real figure, or write down that you do not know, so the person signing for it is choosing with their eyes open.