“We have backups” is one of the most reassuring statements in infrastructure work, and one of the most dangerous when left untested. A backup file may prove that some data existed at a point in time. It does not prove that an incident will be detected, that impact can be contained, that dependencies can be restored in order, or that the recovered business will work.
Recovery is a path through detection, containment, restoration, validation, and review. Missing any stage can leave a team with a running server and an unusable service.
Separate backup, recovery, and continuity
| Capability | Core question | How to verify |
|---|---|---|
| Backup | Are independent copies of data and configuration available? | Check completion, integrity, retention, and permissions |
| Recovery | Can the system be rebuilt within an acceptable time? | Perform an actual restore in isolation |
| Continuity | How is business impact reduced during recovery? | Exercise maintenance, degradation, failover, and communication |
Use RPO and RTO to replace “as soon as possible”
RPO describes the maximum acceptable data-loss window. An hourly form backup may lose nearly one hour of submissions in the worst case. RTO describes the acceptable time from incident to restoration of the critical service.
Smaller targets have cost: lower RPO may require more frequent backup or replication, while lower RTO requires prepared environments, automation, and stronger on-call coverage. Classify business paths instead of demanding zero loss and zero interruption everywhere.
Detect: alerts must lead to action
A useful alert includes time, affected service, current state, recent change, and a route to logs. Divide signals across entry, service health, resource capacity, and synthetic business checks. DNS and TLS are entry signals; process exit and error rate are service signals; disk and memory are capacity; failed form delivery is a business signal.
Contain: prevent expansion before chasing full restoration
Containment may pause releases, enable maintenance mode, rate-limit traffic, revoke credentials, or isolate a node. Preserve logs and snapshots. Cleaning aggressively before collecting evidence can make root-cause analysis impossible.
Recover: order is often more important than speed
Domains, certificates, applications, databases, object storage, mail, and external APIs have ordering dependencies. Opening traffic before database migration completes can create inconsistent writes; starting an application with invalid credentials can produce a healthy process that cannot serve users.
| Order | Confirm | Completion signal |
|---|---|---|
| 1. Entry | Network, domain, TLS, access control | Safe access in isolation |
| 2. Data | Backup point, integrity, migration version | Reads and constraints succeed |
| 3. Application | Release, configuration, secrets, dependencies | Health checks and logs are normal |
| 4. Business journey | Login, forms, mail, downloads, playback | End-to-end operation succeeds |
| 5. Traffic | Monitoring, capacity, rollback point | Metrics remain stable under observation |
Validate through a real user journey
Open ports and live processes are only technical signals. Submit controlled forms, verify downstream delivery, retrieve protected files, test authentication, and compare the restored data time with the agreed RPO. Keep a rollback point while validation is still underway.
Review the system, not only the person
A high-quality review separates facts from assumptions and records the timeline, earliest signal, critical decisions, successful actions, failed attempts, and communication points. “Operator error” is not a root cause until the team understands why validation, permission limits, or automation did not prevent it.
How to run an executable recovery drill
- Scope: choose one system and one failure hypothesis rather than every disaster at once.
- Protect production: restore in isolation where possible and define explicit stop conditions.
- Hide the answer: let operators follow the runbook without constant coaching from its author.
- Measure: record time to detect, gain access, restore data, start the application, and validate the business.
- Correct: update missing credentials, stale paths, screenshots, and hidden dependencies immediately.
Repeat the drill after material migrations, permission changes, database upgrades, or backup redesign. Recovery should not depend on emergency heroics. It is an engineering capability that can be designed, practiced, measured, and improved.