Automated Failover & HA Architecture

The Zero-Downtime Imperative

Indian users expect applications to be available 24/7. Whether they are making UPI payments at midnight, shopping during flash sales, or streaming cricket matches during IPL season, downtime is not acceptable. Automated failover and high availability architectures ensure your services survive component failures without human intervention.

This guide covers the key components, patterns, and implementation strategies for building self-healing infrastructure.

Core Concepts

High Availability (HA)

High availability means your system remains operational despite component failures. It is measured in nines:

99.9 percent = 8.76 hours downtime per year
99.99 percent = 52.6 minutes downtime per year
99.999 percent = 5.26 minutes downtime per year

Automated Failover

Automated failover is the mechanism that detects a failure and redirects traffic to healthy components without human intervention. It is the engine that makes high availability possible.

Health Checks

Health checks are the foundation of automated failover. They continuously verify that each component is functioning correctly.

Architecture Pattern 1: Load Balancer with Health Checks

The most common HA pattern uses a load balancer to distribute traffic across multiple servers:

Internet
    │
    ▼
┌──────────────┐
│ Load Balancer │  ← Health checks every 10s
│  (HAProxy /   │
│   Nginx)      │
└──────┬───────┘
       │
  ┌────┼────┐
  ▼    ▼    ▼
┌───┐┌───┐┌───┐
│S1 ││S2 ││S3 │  ← Application servers
└───┘└───┘└───┘

**HAProxy configuration for automated failover:**

global
    maxconn 50000

frontend http_front bind *:80 bind *:443 ssl crt /etc/ssl/certs/app.pem default_backend app_servers

backend app_servers balance roundrobin option httpchk GET /health http-check expect status 200 server app1 10.0.1.10:8080 check inter 10s fall 3 rise 2 server app2 10.0.1.11:8080 check inter 10s fall 3 rise 2 server app3 10.0.1.12:8080 check inter 10s fall 3 rise 2 # Backup server only used when all primary servers are down server app_backup 10.0.2.10:8080 check backup ```

**Nginx configuration for health check-based failover:**

upstream app_backend {
    server 10.0.1.10:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=30s;
    server 10.0.2.10:8080 backup;
}

server { listen 443 ssl; location / { proxy_pass http://app_backend; proxy_next_upstream error timeout http_500 http_502 http_503; proxy_connect_timeout 5s; proxy_read_timeout 30s; } } ```

Architecture Pattern 2: Database High Availability

Database failover is more complex because of state:

PostgreSQL with Patroni

# Patroni configuration for 3-node cluster
scope: pg-cluster
namespace: /db/postgres/

restapi: listen: 0.0.0.0:8008 authentication: username: patroni password: secure_password

bootstrap: dcs: ttl: 30 loop_wait: 10 retry_timeout: 10 maximum_lag_on_failover: 1048576 synchronous_mode: true postgresql: use_pg_rewind: true parameters: max_connections: 200 shared_buffers: 4GB wal_level: replica max_wal_senders: 5 ```

Patroni automatically: - Detects primary failure - Promotes the most up-to-date standby - Reconfigures other standbys to follow the new primary - Updates the service endpoint via etcd or Consul

MySQL with Group Replication and MySQL Router

-- MySQL Router configuration for automatic failover
[metadata_cache:production]
router_id=1
bootstrap_server_addresses=mysql://node1:3306,mysql://node2:3306,mysql://node3:3306
user=mysql_router_user

[routing:production_rw] bind_address=0.0.0.0:6446 destinations=metadata-cache://production/?role=PRIMARY routing_strategy=first-available

[routing:production_ro] bind_address=0.0.0.0:6447 destinations=metadata-cache://production/?role=SECONDARY routing_strategy=round-robin-with-fallback ```

Architecture Pattern 3: Kubernetes Self-Healing

Kubernetes provides built-in self-healing through its control plane:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    spec:
      containers:
      - name: web-app
        image: app:v2.1
        resources:
          requests:
            cpu: 500m
            memory: 512Mi
          limits:
            cpu: 1000m
            memory: 1Gi
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 10
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 3
        startupProbe:
          httpGet:
            path: /health/startup
            port: 8080
          failureThreshold: 30
          periodSeconds: 10

Kubernetes automatically: - Restarts containers that fail liveness probes - Removes pods from service endpoints when readiness probes fail - Reschedules pods to healthy nodes when a node goes down - Maintains the desired replica count

Architecture Pattern 4: Multi-Layer HA

Production systems typically combine multiple HA patterns:

Internet
    │
    ▼
┌────────────────┐
│ Global DNS with │ ← Geographic routing + health checks
│ failover        │
└────────┬───────┘
         │
    ┌────┴────┐
    ▼         ▼
┌────────┐┌────────┐
│Region 1││Region 2│
│        ││        │
│ ┌─────┐││ ┌─────┐│
│ │  LB │││ │  LB ││  ← L7 load balancer
│ └──┬──┘││ └──┬──┘│
│    │   ││    │   │
│ ┌──┴──┐││ ┌──┴──┐│
│ │App x3│││ │App x3││  ← Multiple app instances
│ └──┬──┘││ └──┬──┘│
│    │   ││    │   │
│ ┌──┴──┐││ ┌──┴──┐│
│ │ DB  │││ │ DB  ││  ← HA database cluster
│ │cluster│││ │cluster││
│ └─────┘││ └─────┘│
└────────┘└────────┘

Implementing Health Checks

A good health check verifies that the application can actually serve requests:

# Flask health check endpoint
@app.route('/health')
def health():
    checks = {}
    
    # Check database connectivity
    try:
        db.session.execute('SELECT 1')
        checks['database'] = 'ok'
    except Exception as e:
        checks['database'] = f'error: {str(e)}'
    
    # Check Redis connectivity
    try:
        redis_client.ping()
        checks['cache'] = 'ok'
    except Exception as e:
        checks['cache'] = f'error: {str(e)}'
    
    # Check disk space
    disk_usage = psutil.disk_usage('/')
    checks['disk'] = 'ok' if disk_usage.percent < 90 else 'warning'
    
    # Overall status
    all_ok = all(v == 'ok' for v in checks.values())
    status_code = 200 if all_ok else 503
    
    return jsonify({'status': 'healthy' if all_ok else 'degraded', 'checks': checks}), status_code

Avoiding Common Pitfalls

Split-Brain

When both primary and standby think they are active, you get split-brain. Prevent it with: - Fencing mechanisms (STONITH — Shoot The Other Node In The Head) - Quorum-based decision making - Consensus protocols (Raft, Paxos)

Cascading Failures

A failing health check causing load balancer to drain all servers: - Set appropriate thresholds (3 failures, not 1) - Implement gradual traffic shifting - Use circuit breaker patterns in application code

Flapping

A server repeatedly going in and out of rotation: - Use hysteresis (different thresholds for up vs down) - Require consecutive successes before marking a server healthy - Implement exponential backoff on health check retries

Monitoring HA Infrastructure

# Prometheus alerting rules for HA
groups:
- name: ha_alerts
  rules:
  - alert: ServiceDegraded
    expr: up{job="app"} < 3
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "Application running with reduced capacity"
      
  - alert: FailoverTriggered
    expr: increase(failover_total[5m]) > 0
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: "Automated failover has been triggered"

Conclusion

High availability is not a single technology but an architecture. Combine load balancers with health checks, database replication with automated failover, and Kubernetes self-healing with multi-region distribution. Test your failover regularly, monitor your HA components, and continuously improve your architecture based on real incidents and drill results.

Automated Failover and High Availability: Architectures That Keep You Online