OneNym.idDiscuss deployment
English

THE HARDENED STACK / Chapter 15

Disaster Recovery and Business Continuity

If ransomware hit the file shares tomorrow, how long until the business works again, and how much data would be lost? This chapter turns those two guesses into agreed numbers, a written recovery order and a plan that has been tested.

This page summarises the chapter. The full text is in the book.

RTO and RPO are business decisions

The chapter begins the morning after Meridian's account compromise, when Daniel Okafor asks Tom Brennan how long it would take to take orders again after a ransomware attack, and how much data would be lost. The honest answer, somewhere between a day and a week, is the problem the chapter sets out to fix. The author treats disaster recovery and business continuity together for small teams and stresses that most of the work is thinking rather than building, so it should not wait for a quiet week.

Recovery Time Objective (RTO) is how long a service can be down before the impact is unacceptable. Recovery Point Objective (RPO) is how much data, measured in time, can be lost; an RPO of one hour means backing up or replicating at least hourly. IT cannot set these alone because it does not own the consequences, so its role is to translate options into costs and let the business choose. Asked in the abstract, people say zero and zero; offered two or three priced options, they choose realistically. The numbers should be set per service, sorted into tiers, written down and signed. The book also warns that time spent recognising and declaring a disaster comes out of the RTO, so name who may invoke the plan, and keep RTO well inside the maximum tolerable downtime.

Strategies, costs and why replication is not backup

Four strategies trade money for speed. Backup and restore, rebuilding infrastructure and restoring data, is cheapest and slowest, and entirely sufficient for services that tolerate hours or days. Pilot light keeps a minimal core such as a replicating database running and scales up on demand. Warm standby runs a smaller but working copy so failover takes minutes. Hot standby or active-active runs full capacity in two places and is usually reserved for one or two services that cannot tolerate minutes of downtime.

For Microsoft 365 and other SaaS, the provider largely handles infrastructure availability, but under the shared-responsibility model your data, configuration and recovery from your own mistakes or a tenant compromise remain yours. Regional redundancy will not bring back a long-deleted mailbox or a SharePoint site encrypted through a compromised account. For advanced readers, the book adds that cost rises steeply as RTO approaches zero, that synchronous replication adds latency to every normal write, and that replication faithfully copies corruption and ransomware to the standby. A sound design pairs replication for availability with immutable, air-gapped, point-in-time backups for integrity.

Dependency mapping and the recovery-order runbook

Recovery means bringing many services back in the right order. Identity comes before the applications that authenticate against it, DNS before almost everything, databases before the application tier. The book recommends writing down what each critical service needs, including outside dependencies such as payment processors, external APIs, email delivery or the DNS registrar, and grouping services into waves. Wave zero is the foundation: networking, DNS, identity, secrets and key management, and certificate services. At Meridian a whiteboard session placed the orders database, ordering portal and finance system in wave one, loyalty and warehouse systems in wave two, and the wiki and reporting in wave three; the surprise was how much sat in wave zero.

The runbook turns the map into numbered steps with a health check between waves, such as confirming an administrator can sign in and read a test secret before any application is restored. It must be stored off-site and printed, because a recovery runbook kept only on the file share being recovered is useless. After recovery the team records the RTO and RPO actually achieved against target, which gives a concrete basis for either investing or revising the target.

Ransomware: rebuild clean rather than restore fast

Ransomware breaks the old assumption that the most recent backup is the best one. Attackers are often present for weeks before triggering, so recent backups may contain their tooling, and they target backups first. The recovery sequence the book describes is to contain and eradicate first, identify a known-clean recovery point that may be older, rebuild the platform from code and golden images rather than whole-system snapshots, restore data into that platform while scanning it, and rotate every credential and secret the attacker could have touched. Pressure to restore immediately is strong, but reinfection costs a second recovery that is harder than the first. Recovery also has a tail of days or weeks of heightened monitoring.

Rebuilding from code is the core principle: the platform is recreated from infrastructure as code such as Terraform, Bicep, ARM or CloudFormation plus configuration management, secrets are issued fresh from the secrets store, and only data is restored from backup. This discards any changes an attacker or operator made by hand. The same discipline also makes everyday changes safer, because a bad change becomes a commit you can revert.

Testing: an untested plan is a hope

Untested plans fail on unreadable backups, references to decommissioned systems, missing recovery credentials and unmapped dependencies. The book describes two levels of testing. Tabletop exercises walk through a scenario such as a regional outage without touching production, at least a couple of times a year and after major architecture changes. Real failover tests restore a service into the DR target and prove it works, starting with one non-critical service in a sandbox and widening each cycle. The checklist covers defining scope and success criteria, notifying stakeholders, following the runbook as written and logging deviations, measuring actual RTO and RPO, smoke-testing, rolling back, and updating the runbook and dependency map from the findings.

KEY POINTS

Key points

  • Set RTO and RPO for each critical service with the business and get the numbers signed.
  • Sort services into tiers and match each tier to a DR strategy you can afford.
  • Map dependencies and group services into recovery waves, starting with network, DNS, identity and secrets.
  • Write a recovery-order runbook with health-check gates and keep printed and off-site copies.
  • Keep immutable, air-gapped backups separate from replication, and define the platform as code.
  • Run a tabletop exercise this quarter and a real failover test this year, and update the runbook from what breaks.

QUESTIONS

Common questions

What is the difference between RTO and RPO?
RTO, the Recovery Time Objective, is how long a service can be down before the impact becomes unacceptable. RPO, the Recovery Point Objective, is how much data you can afford to lose, measured as time between the last good recovery point and the failure. An RPO of one hour, for example, means backing up or replicating at least every hour. Both should be set per service with the business, not by IT alone.
Who should decide RTO and RPO targets?
The business should decide them, because it owns the consequences of downtime and data loss; IT's role is to translate each option into cost. The Hardened Stack recommends presenting two or three concrete options with their prices rather than asking in the abstract, since people asked without a price tend to answer zero and zero. The agreed numbers should be written down and signed.
Is replication enough for disaster recovery?
No. Replication protects against infrastructure failure, but it copies corruption and ransomware encryption to the standby just as faithfully as good data. A sound design pairs replication for availability with immutable, air-gapped, point-in-time backups for integrity. Ask of any architecture which disaster it does not survive, and cover that gap with another layer.
Does Microsoft 365 or the cloud provider handle backup for us?
The provider keeps the service running and handles infrastructure availability, but under the shared-responsibility model your data, your configuration, and recovery from your own mistakes or a compromise of your tenant remain your responsibility. Regional redundancy does not recover a mailbox deleted long ago or a SharePoint site encrypted through a compromised account. Being in the cloud is not the same as being backed up.
In what order should services be restored after a disaster?
Restore in waves based on a dependency map. Wave zero is the foundation: networking, DNS, identity, secrets and key management, and certificate services. Next come the core data stores and business-critical services, then important but tolerable services, then everything else. Confirm each wave is healthy, for example that an administrator can sign in and read a test secret, before starting the next.
How should we recover from ransomware without reinfecting?
Contain and eradicate the attacker first, then choose a known-clean recovery point, which may be older than the latest backup. Rebuild the platform from infrastructure as code and golden images instead of restoring whole-system snapshots, restore only data into it while scanning, and rotate every credential and secret the attacker could have reached. Keep heightened monitoring running for some time after the business is back.