Mastering systemd: Linux Service Management for Cloud Server Administrators

systemd is the init system and service manager on virtually all modern Linux distributions, including Ubuntu, CentOS, Debian, and RHEL. If you manage servers on ServerRaja, understanding systemd is non-negotiable — it controls how services start, stop, restart, and recover. This guide covers everything from basic service management to writing custom unit files and setting resource limits.

Basic Service Management

The `systemctl` command is your primary interface for managing services. Here are the essential operations:

# Start a service
systemctl start nginx

# Stop a service systemctl stop nginx

# Restart a service (stops then starts) systemctl restart nginx

# Reload configuration without dropping connections systemctl reload nginx

# Check service status with recent logs systemctl status nginx

# Enable service to start at boot systemctl enable nginx

# Disable service from starting at boot systemctl disable nginx

# Check if a service is enabled systemctl is-enabled nginx

# Check if a service is active (running) systemctl is-active nginx ```

Listing and Filtering Services

Managing a busy ServerRaja server means dealing with dozens of services. systemd provides powerful filtering:

# List all running services
systemctl list-units --type=service --state=running

# List all failed services (great for troubleshooting) systemctl --failed

# List all services that start at boot systemctl list-unit-files --type=service --state=enabled

# List all services with their status systemctl list-units --type=service

# Check dependencies of a service systemctl list-dependencies nginx ```

Understanding Unit Files

Every service is defined by a unit file. These files are stored in three locations:

• `/usr/lib/systemd/system/` — Package-installed unit files (do not edit)
• `/etc/systemd/system/` — Administrator overrides (your customizations go here)
• `/run/systemd/system/` — Runtime units (temporary, cleared on reboot)

When a unit file exists in multiple locations, the one in `/etc/systemd/system/` takes priority. Never edit files in `/usr/lib/systemd/system/` directly — your changes will be overwritten on package updates.

Creating a Custom Service

Let's say you have a Node.js application on your ServerRaja VPS that you want to manage as a systemd service. Create a unit file:

sudo nano /etc/systemd/system/myapp.service

[Unit]
Description=My Node.js Application
Documentation=https://yourdomain.com/docs
After=network.target
Wants=network-online.target

[Service] Type=simple User=deployer Group=deployer WorkingDirectory=/opt/myapp ExecStart=/usr/bin/node /opt/myapp/server.js ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure RestartSec=5 StartLimitBurst=5 StartLimitIntervalSec=60

# Environment variables Environment=NODE_ENV=production Environment=PORT=3000 EnvironmentFile=-/opt/myapp/.env

# Security hardening NoNewPrivileges=true ProtectSystem=strict ProtectHome=true ReadWritePaths=/opt/myapp/data /var/log/myapp PrivateTmp=true

# Resource limits LimitNOFILE=65535 LimitNPROC=4096 CPUQuota=200% MemoryMax=2G

# Logging StandardOutput=journal StandardError=journal SyslogIdentifier=myapp

[Install] WantedBy=multi-user.target ```

Then enable and start it:

# Reload systemd to pick up the new unit file
systemctl daemon-reload

# Enable and start systemctl enable myapp systemctl start myapp

# Check status systemctl status myapp

# View logs journalctl -u myapp -f ```

Resource Management with systemd

systemd integrates with cgroups to enforce resource limits. This is crucial on a cloud VPS where resources are shared:

# View current resource usage of a service
systemctl status myapp
systemd-cgtop

# Set temporary resource limits (until reboot) systemctl set-property myapp CPUQuota=150% systemctl set-property myapp MemoryMax=1G

# Make limits permanent by adding to the unit file [Service] section: # CPUQuota=150% # MemoryMax=1G # IOWeight=500 # TasksMax=512 ```

Timers: The Modern Cron Alternative

systemd timers replace cron for many scheduled tasks, offering better logging, dependency management, and resource control:

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily backup timer

[Timer] OnCalendar=*-*-* 02:00:00 Persistent=true RandomizedDelaySec=900

[Install] WantedBy=timers.target

# /etc/systemd/system/backup.service [Unit] Description=Run daily backup

[Service] Type=oneshot ExecStart=/opt/scripts/backup.sh User=root ```

systemctl enable backup.timer
systemctl start backup.timer

# List all active timers systemctl list-timers --all ```

Troubleshooting Services

When a service fails on your ServerRaja server, systematic troubleshooting saves time:

# Step 1: Check service status for immediate clues
systemctl status myapp

# Step 2: View detailed logs journalctl -u myapp --since "1 hour ago" --no-pager

# Step 3: Check for OOM (out of memory) kills dmesg | grep -i "oom\|killed"

# Step 4: Validate the unit file syntax systemd-analyze verify /etc/systemd/system/myapp.service

# Step 5: Check boot performance systemd-analyze blame systemd-analyze critical-chain

# Step 6: Debug startup sequence systemd-analyze plot > boot.svg ```

Masking and Unmasking Services

Sometimes you need to completely prevent a service from being started, even as a dependency:

# Mask a service (creates symlink to /dev/null)
systemctl mask cups

# Unmask to restore normal operation systemctl unmask cups ```

This is useful on headless ServerRaja servers where you do not need print services, Bluetooth, or other desktop-oriented daemons. Masking them frees resources and reduces attack surface.

Key Takeaways

• **Use `systemctl enable`** to make services start at boot and `systemctl status` (not just `is-active`) to see recent logs and diagnose why a service failed.
• **Write custom unit files** with `Restart=on-failure`, `RestartSec=5`, and resource limits (`MemoryMax`, `CPUQuota`) to make services self-healing and prevent runaway processes.
• **systemd timers replace cron** for time-based tasks — they offer logging via `journalctl`, dependency ordering, and don't require a running daemon.
• **`journalctl` queries** (by unit, time range, priority) are faster and more powerful than parsing log files — use `-f` to tail and `--since`/`--until` for time-bounded investigation.
• **Always run `systemctl daemon-reload`** after editing unit files — forgetting this is the most common reason changes don't take effect.