Linux IDS/IPS Setup Guide

Why You Need an IDS/IPS

Firewalls control what traffic enters and leaves your network, but they cannot inspect the content of allowed connections. An Intrusion Detection System (IDS) monitors network traffic and system activity for known attack signatures and behavioral anomalies. An Intrusion Prevention System (IPS) goes a step further by automatically blocking detected threats.

Without an IDS/IPS, an attacker who exploits a vulnerability in your web application -- through allowed HTTP/HTTPS traffic -- can operate undetected for weeks or months. Studies show that the average time to detect a breach without monitoring tools exceeds 200 days.

Option 1: Suricata -- High-Performance Network IDS/IPS

Suricata is an open-source, high-performance network IDS/IPS that processes multi-gigabit traffic using multi-threading. It is the modern successor to Snort with better performance and native support for multiple rule formats.

Installation

# Ubuntu
sudo apt install suricata
sudo systemctl enable suricata

# CentOS sudo dnf install epel-release sudo dnf install suricata sudo systemctl enable suricata ```

Configuration

Edit `/etc/suricata/suricata.yaml`:

vars:
  address-groups:
    HOME_NET: "[10.10.0.0/16, 192.168.1.0/24]"
    EXTERNAL_NET: "!$HOME_NET"
  port-groups:
    HTTP_PORTS: "80,8080,8000"
    HTTPS_PORTS: "443"
    SSH_PORTS: "2222"

outputs: - eve-log: enabled: yes filetype: regular filename: /var/log/suricata/eve.json types: - alert - http - dns - tls - files ```

Updating Rules

Suricata uses rule sets to identify threats. Update rules with `suricata-update`:

sudo suricata-update
sudo suricata-update list-sources
sudo suricata-update enable-source et/open
sudo suricata-update
sudo systemctl restart suricata

The ET/Open (Emerging Threats Open) rule set is free and covers thousands of known attack signatures.

Running Suricata in IPS Mode

To move from detection (IDS) to prevention (IPS), configure Suricata with NFQ:

sudo iptables -I FORWARD -j NFQUEUE --queue-num 0
sudo iptables -I INPUT -j NFQUEUE --queue-num 0
sudo suricata -c /etc/suricata/suricata.yaml -q 0

In IPS mode, Suricata drops packets matching drop rules and logs the event. This requires careful tuning to avoid blocking legitimate traffic.

Option 2: OSSEC -- Host-Based Intrusion Detection

While Suricata monitors network traffic, OSSEC focuses on host-level activity: file integrity, log analysis, rootkit detection, and active response.

wget https://github.com/ossec/ossec-hids/archive/3.7.0.tar.gz
tar -xzf 3.7.0.tar.gz
cd ossec-hids-3.7.0
sudo ./install.sh

Choose "local" installation mode. Edit `/var/ossec/etc/ossec.conf`:

<ossec_config>
  <syscheck>
    <frequency>3600</frequency>
    <directories check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
    <directories check_all="yes">/bin,/sbin</directories>
  </syscheck>
  <rootcheck>
    <system_audit>/var/ossec/etc/rootcheck/rootkit_files.txt</system_audit>
    <frequency>86400</frequency>
  </rootcheck>
  <global>
    <email_notification>yes</email_notification>
    <email_to>[email protected]</email_to>
  </global>
</ossec_config>

OSSEC performs four critical functions:

1. **File Integrity Monitoring**: Detects unauthorized changes to system files. 2. **Log Analysis**: Parses logs to detect brute-force attempts and privilege escalation. 3. **Rootkit Detection**: Scans for known rootkit signatures and suspicious kernel modules. 4. **Active Response**: Automatically blocks offending IPs using firewall rules.

Enable automatic blocking of brute-force attackers:

<active-response>
  <command>host-deny</command>
  <location>local</location>
  <level>6</level>
  <timeout>3600</timeout>
</active-response>

Combining Network and Host-Based Detection

The most effective security posture combines both approaches:

**Network (Suricata/Snort)**: Detects port scans, exploit attempts, C2 traffic, data exfiltration.
**Host (OSSEC)**: Detects file tampering, privilege escalation, log anomalies, rootkits.

Deploy Suricata at the network perimeter and OSSEC on every server for comprehensive coverage.

Analyzing Alerts with the Elastic Stack

Suricata's EVE JSON output integrates seamlessly with the Elastic Stack:

# /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: log
    paths:
      - /var/log/suricata/eve.json
    json.keys_under_root: true

output.elasticsearch: hosts: ["localhost:9200"] ```

Import the pre-built Suricata dashboards into Kibana for real-time visualization of alerts, protocol distribution, and geographic attack origins.

Tuning to Reduce False Positives

Every IDS/IPS generates false positives initially. Tune your deployment by:

1. **Running in IDS-only mode** for the first two weeks to understand your traffic patterns. 2. **Suppressing known-good patterns** in `suricata.yaml` using threshold configurations. 3. **Whitelisting internal monitoring tools** that generate automated requests. 4. **Reviewing alerts daily** during the initial deployment period. 5. **Graduating to IPS mode** only after false positives are below an acceptable threshold.

Conclusion

Intrusion detection and prevention are essential components of a mature security posture. Suricata provides high-performance network-level detection, while OSSEC covers host-level threats through file integrity monitoring and log analysis. Together, they form a comprehensive detection layer that catches threats firewalls miss. At ServerRaja, we recommend deploying IDS/IPS on every production server -- especially those handling customer data.

Setting Up Intrusion Detection and Prevention Systems on Linux