Production Application Monitoring: What Should You Measure?

Monitoring is the foundation of reliable operations. Without monitoring, you discover problems only when users report them. Effective monitoring detects issues before they impact users and provides the data needed for diagnosis.

The Four Golden Signals

Based on Google's SRE practices, the four golden signals for monitoring are:

Latency

The time it takes to service a request. Track both successful request latency and error request latency separately. P50 (median), P95, and P99 percentiles are more useful than averages.

Traffic

The demand on your system. For a web application, this is requests per second. For a database, queries per second. Traffic patterns reveal usage trends and capacity needs.

Errors

The rate of failed requests. HTTP 5xx errors indicate server-side failures. HTTP 4xx errors may indicate client issues. Track error rate as a percentage of total traffic.

Saturation

How full your resources are. CPU at 90%, memory at 85%, disk at 80%, network at 70%. Saturation metrics predict when you will run out of capacity.

Infrastructure Metrics

CPU

• Overall utilization (user + system)
• Per-core utilization (identify imbalanced load)
• Iowait (indicates disk bottleneck)
• Load average (1, 5, 15 minute)

Memory

• Total usage percentage
• Available memory (the critical metric)
• Swap usage (any swap = insufficient RAM)
• Per-process memory consumption

Disk

• Filesystem usage percentage
• IOPS utilization
• Read/write latency
• Disk queue depth

Network

• Bandwidth utilization
• Packet rates
• Errors and drops
• Connection counts

Application Metrics

• Request rate (requests/second)
• Response time distribution (P50, P95, P99)
• Error rate (errors/total requests)
• Active connections/sessions
• Queue lengths (if using job queues)
• Cache hit rates

Database Metrics

• Active connections
• Query latency (average and percentiles)
• Cache hit ratio (should be > 99%)
• Lock waits and deadlocks
• Replication lag
• Transaction rate
• Slow query count

Logging

Structured logging provides context for debugging: - Application logs: request/response details, errors, warnings - Access logs: who accessed what and when - System logs: kernel messages, service status changes - Security logs: authentication events, authorization failures

Alerting Strategy

Alert on symptoms, not causes: - Alert on high error rates (symptom) rather than high CPU (cause) - Alert on user-facing latency (symptom) rather than slow queries (cause) - Set thresholds that indicate actionable problems - Avoid alert fatigue: too many alerts leads to ignoring all alerts

Alert Severity Levels

• Critical: immediate response required (service down, data loss risk)
• Warning: response within hours (approaching capacity, degraded performance)
• Informational: review during business hours (minor anomalies, trend changes)

Monitoring Tools

• Prometheus + Grafana: metrics collection and visualization
• ELK Stack (Elasticsearch, Logstash, Kibana): log aggregation and analysis
• Application Performance Monitoring (APM): detailed application tracing
• Uptime monitoring: external checks that verify service availability

Best Practices

• Monitor from the user's perspective (external uptime checks)
• Monitor from the server's perspective (infrastructure metrics)
• Monitor from the application's perspective (application metrics)
• Correlate metrics across layers for efficient troubleshooting
• Store historical metrics for capacity planning
• Review dashboards regularly, not just during incidents

Comprehensive monitoring is an investment that pays for itself many times over in reduced downtime and faster incident resolution.

Key Takeaways

• **The Four Golden Signals** (latency, traffic, errors, saturation) give you a complete picture of user-facing health with minimal instrumentation.
• **Infrastructure metrics** (CPU, memory, disk, network) catch resource exhaustion early — set alerts at 80% thresholds, not at 100%.
• **Application-level metrics** like request rates, error rates by endpoint, and queue depths reveal business-impacting problems that server stats alone miss.
• **Correlate across layers** — a spike in application errors might be caused by database saturation or memory pressure, not a code bug.
• **Alert severity matters** — use a tiered system (critical/page, warning/notify, info/log) so on-call engineers respond to real incidents, not noise.