Kubernetes Disaster Recovery Guide

Kubernetes Is Not a Magic Shield Against Disasters

Many teams assume that running workloads on Kubernetes automatically provides disaster recovery. While Kubernetes excels at self-healing within a cluster, it does not protect against cluster-wide failures, datacenter outages, or data corruption. You still need a dedicated DR strategy for your Kubernetes workloads.

This guide covers practical approaches to disaster recovery for Kubernetes, from backing up cluster state to implementing multi-cluster failover.

What Needs Protection in Kubernetes?

A Kubernetes environment has several components that each need DR consideration:

**etcd cluster**: The brain of Kubernetes, storing all cluster state
**Persistent volumes**: Application data stored on block or file storage
**Application manifests**: Deployments, services, configmaps, secrets
**Container images**: Your application code packaged as images
**External dependencies**: Databases, message queues, third-party services

Strategy 1: Backup and Restore with Velero

Velero is the most popular open-source tool for Kubernetes backup and disaster recovery:

Installing Velero

# Install Velero with S3-compatible storage
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.7.0 \
  --bucket k8s-dr-backups \
  --backup-location-config region=ap-south-1,s3ForcePathStyle=true,s3Url=https://s3.ap-south-1.amazonaws.com \
  --secret-file ./credentials-velero

Creating Backups

# Backup entire cluster
velero backup create full-backup --include-namespaces '*' --wait

# Backup specific namespace velero backup create production-backup \ --include-namespaces production \ --include-resources '*,pods,persistentvolumeclaims' \ --storage-location default \ --ttl 720h0m0s # Retain for 30 days

# Schedule regular backups velero schedule create daily-production \ --schedule='0 2 * * *' \ --include-namespaces production \ --ttl 720h0m0s ```

Restoring from Backup

# List available backups
velero backup get

# Restore specific backup velero restore create --from-backup production-backup-20250115

# Restore specific namespace to a different cluster velero restore create --from-backup production-backup \ --namespace-mappings production:production-restored \ --restore-volumes=true ```

Velero for Persistent Volume Backup

Velero can snapshot persistent volumes:

# Backup with PV snapshots
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: pv-backup
spec:
  includedNamespaces:
  - production
  defaultVolumesToFsBackup: true  # File-system level backup
  storageLocation: default
  ttl: 720h

Strategy 2: etcd Backup and Recovery

etcd is the single source of truth for your Kubernetes cluster. Regular backups are essential:

Automated etcd Backups

#!/bin/bash
# etcd backup script — run on control plane node

ETCDCTL_API=3 etcdctl snapshot save /backups/etcd/$(date +%Y%m%d_%H%M%S).db \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key

# Verify backup ETCDCTL_API=3 etcdctl snapshot status /backups/etcd/latest.db --write-table

# Upload to object storage aws s3 cp /backups/etcd/ s3://k8s-etcd-backups/ --recursive

# Clean up local backups older than 7 days find /backups/etcd/ -mtime +7 -name '*.db' -delete ```

Restoring etcd

# Restore from snapshot
ETCDCTL_API=3 etcdctl snapshot restore /backups/etcd/snapshot.db \
  --data-dir=/var/lib/etcd-restored \
  --name=control-plane-1 \
  --initial-cluster=control-plane-1=https://10.0.0.1:2380 \
  --initial-advertise-peer-urls=https://10.0.0.1:2380

# Update static pod manifest to use restored data cp /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/manifests/etcd.yaml.bak # Edit etcd.yaml to point --data-dir to /var/lib/etcd-restored ```

Strategy 3: GitOps-Based Recovery

If all your Kubernetes manifests are stored in Git (GitOps approach), recovery is straightforward:

# Restore using ArgoCD
argocd app create production \
  --repo https://github.com/your-org/k8s-manifests.git \
  --path production \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace production \
  --sync-policy automated

# ArgoCD automatically reconciles desired state from Git # argocd app sync production ```

This approach works well because: - Git is your source of truth for application configuration - Re-deploying from Git is deterministic and auditable - No need to backup Kubernetes resources separately - Secrets need separate handling (use sealed-secrets or external-secrets)

Strategy 4: Multi-Cluster Failover

For production workloads that need the highest availability:

Architecture

              ┌─────────────────┐
              │   Global DNS /   │
              │   Traffic Mgr    │
              └────────┬────────┘
                       │
          ┌────────────┼────────────┐
          ▼                         ▼
   ┌──────────────┐        ┌──────────────┐
   │ Cluster       │        │ Cluster       │
   │ (Mumbai)      │  sync  │ (Chennai)     │
   │               │◄──────►│               │
   │ ┌───────────┐ │        │ ┌───────────┐ │
   │ │ App Pods  │ │        │ │ App Pods  │ │
   │ │ DB Cluster│ │        │ │ DB Cluster│ │
   │ └───────────┘ │        │ └───────────┘ │
   └──────────────┘        └──────────────┘

Multi-Cluster Service Mesh

Use a service mesh like Istio for multi-cluster traffic management:

# Istio multi-cluster configuration
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: app-dr
  namespace: production
spec:
  host: app.production.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s

Database Replication Across Clusters

For stateful workloads:

# PostgreSQL cluster with cross-cluster replication
apiVersion: acid.zalan.do/v1
kind: postgresql
metadata:
  name: production-db
spec:
  teamId: platform
  volume:
    size: 100Gi
    storageClass: gp3
  numberOfInstances: 3
  users:
    admin:
    - superuser
    - createdb
  databases:
    production: admin
  postgresql:
    version: '15'
    parameters:
      wal_level: replica
      max_wal_senders: '5'
      hot_standby: 'on'

Strategy 5: Application-Level DR

Some applications handle their own replication and failover:

StatefulSets with Persistent Volumes

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-cluster
spec:
  serviceName: redis-cluster
  replicas: 6
  volumeClaimTemplates:
  - metadata:
      name: redis-data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 50Gi
  template:
    spec:
      containers:
      - name: redis
        image: redis:7
        command:
        - redis-server
        - --cluster-enabled
        - 'yes'
        - --appendonly
        - 'yes'
        volumeMounts:
        - name: redis-data
          mountPath: /data

Disaster Recovery Checklist for Kubernetes

Use this checklist to ensure comprehensive DR coverage:

[ ] etcd backups scheduled every 6 hours
[ ] Velero installed and configured with scheduled backups
[ ] All manifests stored in Git with GitOps deployment
[ ] Persistent volume snapshots enabled
[ ] Secrets backed up separately (sealed-secrets or external vault)
[ ] Container images pushed to a registry accessible from DR site
[ ] DNS failover configured with health checks
[ ] Database replication lag monitored
[ ] DR runbook written and tested
[ ] Failover drill conducted quarterly

Monitoring Kubernetes DR Readiness

# Prometheus rules for Kubernetes DR monitoring
groups:
- name: k8s_dr
  rules:
  - alert: VeleroBackupFailed
    expr: velero_backup_failure_total > 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Velero backup has failed"
      
  - alert: EtcdBackupMissing
    expr: time() - etcd_backup_timestamp > 28800  # 8 hours
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "etcd backup is older than 8 hours"
      
  - alert: PVReplicationLag
    expr: pv_replication_lag_seconds > 300
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Persistent volume replication is lagging"

Conclusion

Kubernetes disaster recovery requires a layered approach: Velero for workload backup and restore, etcd snapshots for cluster state, GitOps for declarative recovery, and multi-cluster architectures for the highest availability. Start with Velero and GitOps, add etcd backups, and evolve toward multi-cluster failover as your needs grow. Always test your recovery procedures and monitor your backup health.

Disaster Recovery for Kubernetes Workloads: A Practical Guide