Linux Log Management Guide

Logs are the eyes and ears of your server. Without proper log management, troubleshooting issues on your ServerRaja cloud server becomes guesswork, and disk space silently fills up until services crash. Modern Linux systems offer two complementary logging systems: journald (structured binary logs) and traditional syslog files with logrotate. Mastering both gives you complete visibility into your server's health.

Understanding the Linux Logging Architecture

On most Linux distributions, including Ubuntu, CentOS, and Debian, two logging systems work side by side:

**journald** (systemd-journald): Collects structured, indexed binary logs from all systemd services, the kernel, and syslog-compatible applications. Fast searching with filtering by time, service, priority, and more.
**rsyslog/syslog**: Writes human-readable text logs to `/var/log/`. Traditional format, easy to process with standard text tools.

Most applications send logs to both simultaneously, giving you the best of both worlds.

Working with journald

The `journalctl` command is your primary tool for querying journald:

# View all logs (oldest first)
journalctl

# View logs in reverse (newest first) journalctl -r

# Follow logs in real-time (like tail -f) journalctl -f

# View logs for a specific service journalctl -u nginx journalctl -u mysql --since today

# Filter by time range journalctl --since "2024-01-15 08:00" --until "2024-01-15 12:00" journalctl --since "1 hour ago" journalctl --since yesterday

# Filter by priority (0=emerg to 7=debug) journalctl -p err # Errors and above journalctl -p warning # Warnings and above

# Filter by boot session journalctl -b0 # Current boot journalctl -b-1 # Previous boot

# View kernel messages only journalctl -k

# Combine filters journalctl -u nginx -p err --since "3 days ago" --no-pager

# View logs for a specific PID journalctl _PID=1234

# Show disk usage of the journal journalctl --disk-usage ```

Configuring journald

Edit `/etc/systemd/journald.conf` to control journal behavior:

[Journal]
# Storage mode: persistent keeps logs across reboots
Storage=persistent

# Maximum disk usage for journal logs SystemMaxUse=500M

# Keep at least this much free space SystemKeepFree=1G

# Maximum size of individual journal files SystemMaxFileSize=50M

# How long to keep logs MaxRetentionSec=30day

# Rate limiting: max entries per service per interval RateLimitIntervalSec=30s RateLimitBurst=10000

# Forward to syslog for text file logging ForwardToSyslog=yes ```

After changes, restart journald:

systemctl restart systemd-journald

# Vacuum old logs to free space immediately journalctl --vacuum-time=30d journalctl --vacuum-size=200M ```

Traditional Log Files

The `/var/log/` directory contains text-based logs:

# Essential log files to know
/var/log/syslog          # General system log (Debian/Ubuntu)
/var/log/messages        # General system log (CentOS/RHEL)
/var/log/auth.log        # Authentication logs (Debian/Ubuntu)
/var/log/secure          # Authentication logs (CentOS/RHEL)
/var/log/kern.log        # Kernel messages
/var/log/dmesg           # Boot-time hardware messages
/var/log/apt/            # Package manager logs (Debian/Ubuntu)
/var/log/yum.log         # Package manager logs (CentOS/RHEL)
/var/log/nginx/          # Nginx access and error logs
/var/log/mysql/          # MySQL/MariaDB logs

Useful Log Analysis Commands

# Count error frequency in the last hour
grep -c "ERROR" /var/log/syslog

# Watch for authentication failures in real-time tail -f /var/log/auth.log | grep "Failed"

# Parse Nginx access logs for top IPs awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# Find slow queries in MySQL log grep "Query_time" /var/log/mysql/slow.log | sort -t= -k2 -rn | head -20

# Check log sizes du -sh /var/log/* | sort -rh | head -20 ```

Configuring logrotate

Without log rotation, log files grow until they fill the disk. The `logrotate` utility automatically rotates, compresses, and deletes old log files.

System-Wide Configuration

Edit `/etc/logrotate.conf`:

# Rotate logs weekly
weekly

# Keep 4 weeks of backups rotate 4

# Create new empty log files after rotation create

# Compress old logs with gzip compress compresscmd /usr/bin/gzip compressext .gz

# Don't compress the most recent rotated file delaycompress

# Include all configs from logrotate.d/ include /etc/logrotate.d ```

Application-Specific Rotation

Create dedicated rotation configs in `/etc/logrotate.d/`:

# /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 deployer deployer
    maxsize 100M
    dateext
    dateformat -%Y%m%d
    sharedscripts
    postrotate
        systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}

Testing logrotate

Always test your configuration before relying on it:

# Dry run — shows what would happen without doing it
logrotate -d /etc/logrotate.d/nginx

# Force rotation now (useful for testing) logrotate -f /etc/logrotate.d/nginx

# Check logrotate status cat /var/lib/logrotate/status ```

Practical Monitoring Tips for ServerRaja Servers

Set up a simple log monitoring script that runs daily via cron:

#!/bin/bash
# /opt/scripts/log-monitor.sh

LOG=/var/log/log-monitor.log

# Check for disk space issues df -h | awk '$5+0 > 80 {print "WARNING: " $0}' >> $LOG

# Check for OOM kills journalctl -k --since "24 hours ago" | grep -i "oom" >> $LOG

# Count failed SSH attempts journalctl -u sshd --since "24 hours ago" | grep -c "Failed" >> $LOG

# Check for service failures systemctl --failed --no-pager >> $LOG ```

Effective log management turns chaos into clarity. With journald for fast queries and logrotate for space management, your ServerRaja server will maintain complete operational history without running out of disk space.

Key Takeaways

**journald is your primary tool** for querying system logs — combine filters by unit, time range, and priority (e.g., `journalctl -u nginx --since "1 hour ago" -p err`) for fast root-cause analysis.
**Configure persistent journald storage** (`Storage=persistent`) and set `SystemMaxUse` to prevent journal logs from filling your disk in production.
**logrotate prevents disk exhaustion** — configure application-specific rotation (daily, compressed, with retention limits) for services like Nginx and MySQL.
**Test logrotate with `logrotate -d`** (dry run) before deploying configs, and use `logrotate -f` to force rotation during testing.
**Write a daily log monitoring script** that checks for disk space issues, OOM kills, failed SSH attempts, and service failures — catching problems in logs beats discovering them from user complaints.

Linux Log Management with journald and logrotate: A Practical Guide