Testing Your Disaster Recovery Plan: How to Run Effective DR Drills

A Plan That Has Never Been Tested Is Not a Plan

You have invested weeks building a disaster recovery plan. You have documented procedures, configured replication, and set up automated failover. But can you actually execute it under pressure? The only way to know is to test it.

DR drills are the difference between a plan that works on paper and one that works in reality. This guide covers how to design, execute, and learn from effective disaster recovery drills.

Types of DR Tests

1. Plan Review (Tabletop Exercise)

The simplest test: gather your team and walk through the DR plan verbally.

**How to run it:** - Schedule 2-3 hours with all stakeholders - Present a disaster scenario - Walk through each step of the DR plan - Ask probing questions at each decision point - Document gaps, ambiguities, and missing steps

**When to use it**: Quarterly, or whenever the plan changes significantly.

**Example scenario**: "A ransomware attack has encrypted all data on the primary Mumbai datacenter at 2 AM on a Saturday. The attackers are demanding ₹50 lakhs. Walk me through what happens next."

2. Component Testing

Test individual components of your DR plan in isolation:

# Test backup restoration
#!/bin/bash
echo "Starting backup restoration test..."
START_TIME=$(date +%s)

# Restore database backup to test instance pg_restore -h test-db-server -d test_db /backups/latest/full_backup.dump

# Verify data integrity psql -h test-db-server -d test_db -c " SELECT COUNT(*) as total_orders FROM orders; SELECT MAX(created_at) as latest_record FROM orders; "

END_TIME=$(date +%s) DURATION=$((END_TIME - START_TIME)) echo "Restoration completed in $DURATION seconds" echo "Actual RTO for database: $DURATION seconds" ```

Test these components individually: - Backup restoration speed - DNS failover time - Replication lag under load - Monitoring and alerting triggering - Communication tool functionality

3. Parallel Testing (Shadow Mode)

Run your DR environment alongside production without affecting live traffic:

• Route a copy of production traffic to the DR environment
• Verify that the DR environment processes requests correctly
• Compare results between production and DR
• Measure the time to bring DR to full capacity

This is the safest way to validate your DR environment without any risk to production.

4. Full Interruption Test

The most realistic test: actually shut down your primary environment and run from DR.

**Warning**: This carries real risk. Schedule during a maintenance window and have rollback procedures ready.

**Procedure:**

1. Notify all stakeholders 2 weeks in advance 2. Verify DR environment health 3. Announce maintenance window to customers 4. Shut down primary services 5. Execute failover procedures 6. Verify functionality from DR environment 7. Run synthetic tests to validate all critical paths 8. Monitor for issues for 1-2 hours 9. Failback to primary 10. Conduct post-mortem

Designing a DR Drill

Define Objectives

Before each drill, be clear about what you want to validate:

• Can we meet our RTO of 30 minutes for the payment system?
• Does our automated failover trigger correctly?
• Can the on-call team execute the runbook without senior engineer help?
• Does our monitoring detect the failure and alert the right people?

Create Realistic Scenarios

Draw from real-world risks specific to Indian infrastructure:

Prepare a Runbook

A DR runbook is a step-by-step guide that any team member can follow:

RUNBOOK: Payment System Failover to Chennai DR

Pre-conditions: - Primary Mumbai payment system is confirmed down - Decision to failover approved by: [Engineering Lead on-call]

Steps: 1. [0 min] Verify primary is truly down (check 3 health endpoints) 2. [2 min] Page the DR team lead 3. [5 min] Execute DNS failover script: /opt/scripts/dns-failover.sh 4. [7 min] Verify DNS propagation: dig payment.example.com 5. [10 min] Scale up DR app servers: kubectl scale deploy/payment --replicas=8 6. [12 min] Verify DR database is promoted: psql -c 'SELECT pg_is_in_recovery()' 7. [15 min] Run smoke tests: /opt/scripts/smoke-tests.sh 8. [20 min] Monitor error rates for 10 minutes 9. [30 min] Declare DR complete, update status page

Rollback: - If DR fails at any step, revert DNS to primary: /opt/scripts/dns-restore.sh ```

Executing the Drill

Pre-Drill Checklist

• All team members briefed on their roles
• Communication channels tested (Slack, phone bridge, email)
• DR environment health verified
• Customer communication templates prepared
• Rollback procedures reviewed
• Monitoring dashboards open

During the Drill

• Assign a timekeeper to track every action and timestamp
• Use a dedicated Slack channel for drill communication
• Record every decision, including the reasoning
• Note anything that goes differently than expected
• Have a "red team" member inject complications ("The standby database is also corrupt" or "The on-call engineer is unreachable")

Post-Drill Activities

After every drill, conduct a thorough retrospective:

**Metrics to capture:** - Time from disaster detection to team mobilization - Time to execute each step in the runbook - Total time to restore service (actual RTO) - Amount of data lost (actual RPO) - Number of steps that required deviation from the runbook

**Questions to answer:** - Did we meet our RTO and RPO targets? - What was the most confusing step? - Were there any tools or access we needed but did not have? - Would a junior engineer have been able to execute this runbook? - What would we do differently next time?

Common DR Drill Failures (and Lessons Learned)

The DNS That Would Not Propagate

A Bengaluru SaaS company discovered during their drill that their DNS TTL was set to 86400 seconds (24 hours). Failover would take up to 24 hours for some users.

**Lesson**: Set DNS TTL to 60 seconds for critical records. Pre-stage DNS changes where possible.

The Expired SSL Certificate

During a drill, the DR environment was fully functional but users saw SSL certificate errors because the DR site used a self-signed certificate.

**Lesson**: Include SSL certificate provisioning in your DR automation. Use wildcard certificates or automated certificate management.

The Missing Database User

The DR database was restored perfectly, but the application could not connect because the database user was not created in the restored instance.

**Lesson**: Include user creation, permissions, and connection strings in your DR automation scripts.

The On-Call Engineer Who Was Asleep

The automated alert fired at 3 AM but the on-call engineer slept through it for 45 minutes.

**Lesson**: Use escalating alerts. If the primary contact does not respond in 5 minutes, escalate to the secondary. Use phone calls, not just Slack notifications, for critical alerts.

Frequency and Cadence

Building a DR Culture

DR drills should not feel like a chore. Build a culture where:

• New team members participate in their first drill within 30 days
• DR readiness is a performance metric for the infrastructure team
• Drill results are shared transparently with leadership
• Failures during drills are treated as learning opportunities, not blame events
• Successful DR responses are celebrated

Conclusion

Regular DR drills transform your disaster recovery plan from a document into a capability. Start with tabletop exercises, graduate to component testing, and work toward annual full-interruption tests. Document everything, learn from every drill, and continuously improve. When a real disaster strikes, your team will respond with confidence instead of panic.