Server Performance Monitoring with Prometheus and Grafana: A Complete Setup

Why Prometheus and Grafana?

Prometheus is the de facto standard for cloud-native monitoring. It scrapes metrics from your servers, stores them in a time-series database, and provides a powerful query language called PromQL. Grafana connects to Prometheus as a data source and renders those metrics into beautiful, interactive dashboards.

Together, they give you deep visibility into your ServerRaja cloud servers without expensive commercial monitoring solutions.

Architecture Overview

The monitoring stack consists of Prometheus Server (collects and stores metrics), Node Exporter (exposes hardware/OS metrics per server), Alertmanager (handles notifications), and Grafana (visualization dashboards).

Installing Prometheus

Set up Prometheus on a dedicated monitoring server:

# Create monitoring user
sudo useradd --no-create-home --shell /bin/false prometheus

# Create directories sudo mkdir -p /etc/prometheus /var/lib/prometheus

# Download and install Prometheus PROM_VERSION="2.50.0" wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz tar xzf prometheus-${PROM_VERSION}.linux-amd64.tar.gz sudo cp prometheus-${PROM_VERSION}.linux-amd64/prometheus /usr/local/bin/ sudo cp prometheus-${PROM_VERSION}.linux-amd64/promtool /usr/local/bin/ sudo cp -r prometheus-${PROM_VERSION}.linux-amd64/consoles /etc/prometheus sudo cp -r prometheus-${PROM_VERSION}.linux-amd64/console_libraries /etc/prometheus

# Set ownership sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus ```

Create the Prometheus configuration:

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files: - "rules/*.yml"

alerting: alertmanagers: - static_configs: - targets: ['localhost:9093']

scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090']

- job_name: 'node-exporter' static_configs: - targets: - 'web1.serverraja.com:9100' - 'web2.serverraja.com:9100' - 'db1.serverraja.com:9100' relabel_configs: - source_labels: [__address__] regex: '(.+):9100' target_label: instance replacement: '${1}'

- job_name: 'nginx-exporter' static_configs: - targets: ['web1.serverraja.com:9113']

- job_name: 'postgres-exporter' static_configs: - targets: ['db1.serverraja.com:9187'] ```

Create a systemd service:

# /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Monitoring
Wants=network-online.target
After=network-online.target

[Service] User=prometheus Group=prometheus Type=simple ExecStart=/usr/local/bin/prometheus \ --config.file=/etc/prometheus/prometheus.yml \ --storage.tsdb.path=/var/lib/prometheus/ \ --web.console.templates=/etc/prometheus/consoles \ --web.console.libraries=/etc/prometheus/console_libraries \ --web.enable-lifecycle \ --storage.tsdb.retention.time=30d

[Install] WantedBy=multi-user.target ```

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus

Installing Node Exporter

Install Node Exporter on every server you want to monitor:

NODE_EXP_VERSION="1.7.0"
wget https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXP_VERSION}/node_exporter-${NODE_EXP_VERSION}.linux-amd64.tar.gz
tar xzf node_exporter-${NODE_EXP_VERSION}.linux-amd64.tar.gz
sudo cp node_exporter-${NODE_EXP_VERSION}.linux-amd64/node_exporter /usr/local/bin/

sudo useradd --no-create-home --shell /bin/false node_exporter

# Create systemd service cat <<EOF | sudo tee /etc/systemd/system/node_exporter.service [Unit] Description=Node Exporter After=network.target

[Service] User=node_exporter ExecStart=/usr/local/bin/node_exporter Restart=always

[Install] WantedBy=multi-user.target EOF

sudo systemctl daemon-reload sudo systemctl enable --now node_exporter ```

Installing Grafana

sudo apt-get install -y apt-transport-https software-properties-common wget
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install grafana
sudo systemctl enable --now grafana-server

Access Grafana at `http://monitoring-server:3000` (default credentials: admin/admin).

Add Prometheus as a data source:

curl -X POST http://admin:admin@localhost:3000/api/datasources \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Prometheus",
    "type": "prometheus",
    "url": "http://localhost:9090",
    "access": "proxy",
    "isDefault": true
  }'

Essential PromQL Queries

Monitor CPU usage:

100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Monitor memory usage:

(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

Monitor disk I/O:

rate(node_disk_read_bytes_total[5m]) + rate(node_disk_written_bytes_total[5m])

Monitor network throughput:

rate(node_network_receive_bytes_total{device="eth0"}[5m]) * 8

Filesystem usage:

(1 - node_filesystem_avail_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes{fstype!="tmpfs"}) * 100

Setting Up Alerts

Create alert rules in `/etc/prometheus/rules/server-alerts.yml`:

groups:
  - name: server-alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 85% for more than 5 minutes (current: {{ $value }}%)"

- alert: HighMemoryUsage expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90 for: 5m labels: severity: critical annotations: summary: "High memory usage on {{ $labels.instance }}"

- alert: DiskSpaceLow expr: (1 - node_filesystem_avail_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes{fstetype!="tmpfs"}) * 100 > 85 for: 10m labels: severity: warning annotations: summary: "Disk space low on {{ $labels.instance }}"

- alert: ServerDown expr: up == 0 for: 1m labels: severity: critical annotations: summary: "Server {{ $labels.instance }} is down" ```

Conclusion

A Prometheus and Grafana monitoring stack gives you enterprise-grade observability at zero software cost. Start by installing Node Exporter on all your ServerRaja servers, configure Prometheus to scrape them, and build Grafana dashboards for your most critical metrics. Add alerting rules to get notified before small problems become big outages.