Log Aggregation and Analysis with ELK Stack: Centralize Your Logs

Why Centralized Log Management?

When you run applications across multiple servers, logging into each server individually to search through log files is inefficient and error-prone. Centralized log management aggregates logs from all your servers into a single searchable system, enabling faster troubleshooting, security auditing, and operational insights.

The ELK stack (Elasticsearch, Logstash, and Kibana) is the most popular open-source solution for log aggregation. Deploying it on ServerRaja cloud servers gives you enterprise-grade log management without the SaaS price tag.

ELK Stack Architecture

The stack consists of three core components:

• **Elasticsearch**: A distributed search and analytics engine that stores and indexes logs
• **Logstash**: A data processing pipeline that collects, parses, and transforms log data
• **Kibana**: A web interface for searching, visualizing, and exploring logs

Modern deployments often add **Filebeat** as a lightweight log shipper that runs on each application server.

Installing Elasticsearch

Set up Elasticsearch on a dedicated server:

# Import the Elasticsearch GPG key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

# Add the repository echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

# Install sudo apt update sudo apt install elasticsearch

# Configure Elasticsearch cat <<EOF | sudo tee /etc/elasticsearch/elasticsearch.yml cluster.name: serverraja-logs node.name: es-node-1 path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 0.0.0.0 http.port: 9200 discovery.type: single-node xpack.security.enabled: true xpack.security.enrollment.enabled: false EOF

# Configure JVM heap (set to 50% of available RAM, max 32GB) cat <<EOF | sudo tee /etc/elasticsearch/jvm.options.d/heap.options -Xms4g -Xmx4g EOF

# Start Elasticsearch sudo systemctl enable --now elasticsearch

# Reset the elastic user password sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic ```

Verify Elasticsearch is running:

curl -k -u elastic:password https://localhost:9200/_cluster/health?pretty

Installing Logstash

sudo apt install logstash

# Create a pipeline configuration for syslog cat <<EOF | sudo tee /etc/logstash/conf.d/syslog-pipeline.conf input { beats { port => 5044 ssl => true ssl_certificate => "/etc/logstash/certs/logstash.crt" ssl_key => "/etc/logstash/certs/logstash.key" } }

filter { if [fields][log_type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:hostname} %{DATA:program}(?:\[%{POSINT:pid}\])?: %{GREEDYDATA:msg}" } } date { match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } }

if [fields][log_type] == "nginx" { grok { match => { "message" => '%{IPORHOST:remote_ip} - %{DATA:user} \[%{HTTPDATE:time}\] "%{WORD:method} %{DATA:url} HTTP/%{NUMBER:ver}" %{NUMBER:code} %{NUMBER:bytes}' } } mutate { convert => { "code" => "integer" "bytes" => "integer" } } geoip { source => "remote_ip" } }

if [fields][log_type] == "application" { json { source => "message" target => "app" } } }

output { elasticsearch { hosts => ["https://localhost:9200"] user => "elastic" password => "password" ssl => true cacert => "/etc/logstash/certs/http_ca.crt" index => "%{[fields][log_type]}-%{+YYYY.MM.dd}" } } EOF

sudo systemctl enable --now logstash ```

Installing Kibana

sudo apt install kibana

# Configure Kibana cat <<EOF | sudo tee /etc/kibana/kibana.yml server.port: 5601 server.host: "0.0.0.0" server.name: "serverraja-kibana" elasticsearch.hosts: ["https://localhost:9200"] elasticsearch.username: "kibana_system" elasticsearch.password: "password" elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/http_ca.crt"] xpack.security.enabled: true EOF

# Set the kibana_system password curl -k -u elastic:password -X POST "https://localhost:9200/_security/user/kibana_system/_password" -H 'Content-Type: application/json' -d '{"password": "password"}'

sudo systemctl enable --now kibana ```

Access Kibana at `http://your-server:5601`.

Installing Filebeat on Application Servers

Deploy Filebeat on each server that generates logs:

sudo apt install filebeat

cat <<EOF | sudo tee /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/syslog - /var/log/auth.log fields: log_type: syslog fields_under_root: false

- type: log enabled: true paths: - /var/log/nginx/access.log - /var/log/nginx/error.log fields: log_type: nginx fields_under_root: false

- type: log enabled: true paths: - /var/log/app/*.log fields: log_type: application fields_under_root: false

output.logstash: hosts: ["logstash-server:5044"] ssl: certificate_authorities: ["/etc/filebeat/certs/ca.crt"] certificate: "/etc/filebeat/certs/client.crt" key: "/etc/filebeat/certs/client.key"

processors: - add_host_metadata: when.not.contains.tags: forwarded - add_cloud_metadata: ~

logging.level: info logging.to_files: true logging.files: path: /var/log/filebeat name: filebeat keepfiles: 7 permissions: 0640 EOF

sudo systemctl enable --now filebeat ```

Managing Elasticsearch Indices

Set up Index Lifecycle Management (ILM) to manage storage growth and automatically delete old data:

# Create an ILM policy that retains logs for 30 days
curl -k -u elastic:password -X PUT "https://localhost:9200/_ilm/policy/log-retention" -H 'Content-Type: application/json' -d '{
  "policy": {
    "phases": {
      "hot": {
        "actions": { "rollover": { "max_size": "50gb", "max_age": "1d" } }
      },
      "warm": {
        "min_age": "7d",
        "actions": { "shrink": { "number_of_shards": 1 }, "forcemerge": { "max_num_segments": 1 } }
      },
      "delete": { "min_age": "30d", "actions": { "delete": {} } }
    }
  }
}'

# Create an index template with the ILM policy curl -k -u elastic:password -X PUT "https://localhost:9200/_index_template/logs-template" -H 'Content-Type: application/json' -d '{ "index_patterns": ["syslog-*", "nginx-*", "application-*"], "template": { "settings": { "index.lifecycle.name": "log-retention", "number_of_shards": 1, "number_of_replicas": 0 } } }' ```

Useful Kibana Searches

Find errors across all logs with `message: "error" OR message: "exception"`, failed SSH logins with `log_type: "syslog" AND message: "Failed password"`, and slow requests with `log_type: "nginx" AND request_time: >1`.

Monitor Elasticsearch disk usage with `curl -k -u elastic:password "https://localhost:9200/_cat/indices?v&s=store.size:desc"` and delete old indices when needed. ILM policies automate this, but manual cleanup is useful during initial growth.

Conclusion

The ELK stack provides powerful centralized log management for your entire ServerRaja cloud infrastructure. Start with Filebeat shipping syslog and Nginx logs to a single-node Elasticsearch cluster. Add application log parsing with Logstash filters, and build Kibana dashboards for common troubleshooting scenarios. Implement ILM policies from the start to manage storage growth, and your team will never need to SSH into individual servers to search logs again.