How to detect intrusions in Linux

How to Detect Intrusions in Linux Linux systems are prime targets for cyber attackers due to their widespread use in servers, cloud environments, and critical infrastructure. Detecting intrusions early is crucial for maintaining system security and preventing data breaches. This comprehensive guide will teach you how to implement effective intrusion detection strategies on Linux systems, from basic log monitoring to advanced security frameworks. Table of Contents 1. [Understanding Linux Intrusion Detection](#understanding-linux-intrusion-detection) 2. [Prerequisites and Requirements](#prerequisites-and-requirements) 3. [System Log Analysis](#system-log-analysis) 4. [Network-Based Intrusion Detection](#network-based-intrusion-detection) 5. [Host-Based Intrusion Detection](#host-based-intrusion-detection) 6. [File Integrity Monitoring](#file-integrity-monitoring) 7. [Real-Time Monitoring Tools](#real-time-monitoring-tools) 8. [Automated Detection Scripts](#automated-detection-scripts) 9. [Common Attack Patterns](#common-attack-patterns) 10. [Troubleshooting and Common Issues](#troubleshooting-and-common-issues) 11. [Best Practices](#best-practices) 12. [Conclusion](#conclusion) Understanding Linux Intrusion Detection Intrusion detection in Linux involves monitoring system activities, network traffic, and file changes to identify unauthorized access attempts, malicious activities, or security policy violations. There are two primary approaches: - Network-based Intrusion Detection Systems (NIDS): Monitor network traffic for suspicious patterns - Host-based Intrusion Detection Systems (HIDS): Monitor individual systems for malicious activities Effective intrusion detection requires a multi-layered approach combining both methods with proper log analysis and real-time monitoring. Prerequisites and Requirements Before implementing intrusion detection mechanisms, ensure you have: System Requirements - Linux system with root or sudo access - Sufficient disk space for log storage (minimum 10GB recommended) - Network connectivity for downloading tools and updates - Basic understanding of Linux command line Software Dependencies ```bash Update system packages sudo apt update && sudo apt upgrade -y # Debian/Ubuntu sudo yum update -y # CentOS/RHEL sudo dnf update -y # Fedora Install essential tools sudo apt install -y rsyslog logwatch fail2ban aide tripwire # Debian/Ubuntu sudo yum install -y rsyslog logwatch fail2ban aide tripwire # CentOS/RHEL ``` User Permissions Ensure your user account has appropriate permissions: ```bash Add user to necessary groups sudo usermod -a -G adm,syslog $USER ``` System Log Analysis System logs are the foundation of intrusion detection. Linux systems generate various logs that contain valuable security information. Understanding Linux Log Files Key log files for intrusion detection: ```bash Authentication logs /var/log/auth.log # Debian/Ubuntu /var/log/secure # CentOS/RHEL System messages /var/log/syslog # General system logs /var/log/messages # System messages Application logs /var/log/apache2/ # Apache web server /var/log/nginx/ # Nginx web server /var/log/mysql/ # MySQL database ``` Analyzing Authentication Logs Monitor failed login attempts and suspicious authentication activities: ```bash Check failed SSH login attempts sudo grep "Failed password" /var/log/auth.log | tail -20 Monitor successful root logins sudo grep "session opened for user root" /var/log/auth.log Check for privilege escalation attempts sudo grep "sudo:" /var/log/auth.log | grep "FAILED" Analyze login patterns sudo awk '/Failed password/ {print $1, $2, $3, $9, $11}' /var/log/auth.log | sort | uniq -c | sort -nr ``` Real-Time Log Monitoring Use `tail` and `watch` commands for real-time monitoring: ```bash Monitor authentication logs in real-time sudo tail -f /var/log/auth.log Watch for specific patterns sudo tail -f /var/log/auth.log | grep --line-buffered "Failed password" Monitor multiple logs simultaneously sudo multitail /var/log/auth.log /var/log/syslog /var/log/apache2/access.log ``` Log Analysis with grep and awk Create powerful one-liners for log analysis: ```bash Find IP addresses with multiple failed attempts sudo awk '/Failed password/ {ip[$(NF-3)]++} END {for (i in ip) if (ip[i] > 5) print ip[i], i}' /var/log/auth.log Detect brute force attacks sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head -10 Monitor unusual user agent strings in web logs sudo awk '{print $12 " " $13 " " $14}' /var/log/apache2/access.log | sort | uniq -c | sort -nr | head -20 ``` Network-Based Intrusion Detection Network-based intrusion detection monitors traffic patterns and identifies malicious network activities. Installing and Configuring Suricata Suricata is a powerful open-source NIDS: ```bash Install Suricata sudo apt install suricata -y # Debian/Ubuntu sudo yum install suricata -y # CentOS/RHEL Configure network interface sudo nano /etc/suricata/suricata.yaml Update rules sudo suricata-update sudo systemctl restart suricata sudo systemctl enable suricata ``` Basic Suricata configuration: ```yaml /etc/suricata/suricata.yaml vars: address-groups: HOME_NET: "[192.168.1.0/24,10.0.0.0/8,172.16.0.0/12]" EXTERNAL_NET: "!$HOME_NET" af-packet: - interface: eth0 cluster-id: 99 cluster-type: cluster_flow ``` Monitoring Network Connections Use netstat and ss to monitor active connections: ```bash Monitor active network connections sudo netstat -tulpn | grep LISTEN Check for suspicious connections sudo ss -tulpn | grep :22 # SSH connections sudo ss -tulpn | grep :80 # HTTP connections Monitor network statistics sudo netstat -i # Interface statistics sudo netstat -s # Protocol statistics ``` Detecting Port Scans Create a script to detect port scanning activities: ```bash #!/bin/bash port_scan_detector.sh LOG_FILE="/var/log/port_scan.log" THRESHOLD=10 Monitor for port scanning patterns sudo netstat -an | grep SYN_RECV | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr > /tmp/connections while read count ip; do if [ $count -gt $THRESHOLD ]; then echo "$(date): Potential port scan detected from $ip ($count connections)" >> $LOG_FILE # Optional: Block IP using iptables # sudo iptables -A INPUT -s $ip -j DROP fi done < /tmp/connections ``` Host-Based Intrusion Detection Host-based intrusion detection focuses on monitoring individual system activities and file changes. Installing and Configuring OSSEC OSSEC is a comprehensive HIDS solution: ```bash Download and install OSSEC wget https://github.com/ossec/ossec-hids/archive/3.7.0.tar.gz tar -xzf 3.7.0.tar.gz cd ossec-hids-3.7.0 sudo ./install.sh ``` Basic OSSEC configuration: ```xml yes localhost ossec@yourdomain.com admin@yourdomain.com 7200 /etc,/usr/bin,/usr/sbin /bin,/sbin 7200 ``` Process Monitoring Monitor running processes for suspicious activities: ```bash List all running processes ps aux | head -20 Monitor process changes watch -n 5 'ps aux --sort=-%cpu | head -20' Check for processes running as root ps aux | grep root | grep -v '\[.*\]' Monitor network processes sudo lsof -i -P -n | grep LISTEN Check for unusual parent-child process relationships ps -eo pid,ppid,cmd --forest ``` User Activity Monitoring Track user activities and sessions: ```bash Check currently logged-in users who w Review user login history last | head -20 lastlog Monitor user commands (if history logging is enabled) sudo find /home -name ".bash_history" -exec grep -l "sudo\|su\|wget\|curl" {} \; Check for unusual user accounts awk -F: '$3 >= 1000 {print $1, $3, $7}' /etc/passwd ``` File Integrity Monitoring File integrity monitoring detects unauthorized changes to critical system files. Using AIDE (Advanced Intrusion Detection Environment) AIDE creates a database of file checksums and monitors changes: ```bash Install AIDE sudo apt install aide -y Initialize AIDE database sudo aideinit Move database to proper location sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db Run integrity check sudo aide --check Update database after legitimate changes sudo aide --update sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db ``` Configure AIDE rules in `/etc/aide/aide.conf`: ```bash Critical system directories /bin FIPSR /sbin FIPSR /usr/bin FIPSR /usr/sbin FIPSR /etc FIPSR Log files (size and timestamps can change) /var/log L Custom rules !/var/log/.* !/tmp/.* !/proc/.* ``` Using Tripwire Tripwire is another powerful file integrity monitoring tool: ```bash Install Tripwire sudo apt install tripwire -y Configure Tripwire sudo tripwire-setup-keyfiles Initialize database sudo tripwire --init Run integrity check sudo tripwire --check Generate report sudo tripwire --check --interactive ``` Custom File Monitoring Script Create a simple file monitoring script: ```bash #!/bin/bash file_monitor.sh WATCH_DIRS=("/etc" "/usr/bin" "/usr/sbin" "/bin" "/sbin") BASELINE_FILE="/var/log/file_baseline.txt" REPORT_FILE="/var/log/file_changes.txt" create_baseline() { echo "Creating file baseline..." > $BASELINE_FILE for dir in "${WATCH_DIRS[@]}"; do find $dir -type f -exec sha256sum {} \; 2>/dev/null >> $BASELINE_FILE done } check_changes() { echo "Checking for file changes..." TEMP_FILE="/tmp/current_files.txt" > $TEMP_FILE for dir in "${WATCH_DIRS[@]}"; do find $dir -type f -exec sha256sum {} \; 2>/dev/null >> $TEMP_FILE done diff $BASELINE_FILE $TEMP_FILE > $REPORT_FILE if [ -s $REPORT_FILE ]; then echo "File changes detected! Check $REPORT_FILE" # Send alert email mail -s "File Integrity Alert" admin@yourdomain.com < $REPORT_FILE fi } case "$1" in baseline) create_baseline ;; check) check_changes ;; *) echo "Usage: $0 {baseline|check}" exit 1 ;; esac ``` Real-Time Monitoring Tools Using fail2ban for Automated Response Fail2ban automatically blocks IP addresses showing malicious behavior: ```bash Install fail2ban sudo apt install fail2ban -y Configure fail2ban sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local sudo nano /etc/fail2ban/jail.local ``` Basic fail2ban configuration: ```ini /etc/fail2ban/jail.local [DEFAULT] bantime = 3600 findtime = 600 maxretry = 3 backend = systemd [sshd] enabled = true port = ssh filter = sshd logpath = /var/log/auth.log maxretry = 3 [apache-auth] enabled = true port = http,https filter = apache-auth logpath = /var/log/apache2/error.log maxretry = 6 [nginx-http-auth] enabled = true port = http,https filter = nginx-http-auth logpath = /var/log/nginx/error.log maxretry = 6 ``` System Resource Monitoring Monitor system resources for anomalies: ```bash CPU and memory monitoring top -b -n 1 | head -20 htop Disk usage monitoring df -h du -sh /var/log/* I/O monitoring iostat -x 1 5 iotop Network monitoring iftop nethogs ``` Creating Custom Monitoring Scripts Develop automated monitoring scripts: ```bash #!/bin/bash system_monitor.sh LOG_FILE="/var/log/system_monitor.log" ALERT_EMAIL="admin@yourdomain.com" Function to log with timestamp log_message() { echo "$(date '+%Y-%m-%d %H:%M:%S'): $1" >> $LOG_FILE } Check CPU usage check_cpu() { CPU_USAGE=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1) if (( $(echo "$CPU_USAGE > 90" | bc -l) )); then log_message "HIGH CPU USAGE: $CPU_USAGE%" echo "High CPU usage detected: $CPU_USAGE%" | mail -s "CPU Alert" $ALERT_EMAIL fi } Check memory usage check_memory() { MEM_USAGE=$(free | grep Mem | awk '{printf("%.2f", $3/$2 * 100.0)}') if (( $(echo "$MEM_USAGE > 90" | bc -l) )); then log_message "HIGH MEMORY USAGE: $MEM_USAGE%" echo "High memory usage detected: $MEM_USAGE%" | mail -s "Memory Alert" $ALERT_EMAIL fi } Check disk usage check_disk() { DISK_USAGE=$(df / | tail -1 | awk '{print $5}' | cut -d'%' -f1) if [ $DISK_USAGE -gt 90 ]; then log_message "HIGH DISK USAGE: $DISK_USAGE%" echo "High disk usage detected: $DISK_USAGE%" | mail -s "Disk Alert" $ALERT_EMAIL fi } Check for suspicious processes check_processes() { # Look for processes with unusual names or high resource usage SUSPICIOUS=$(ps aux --sort=-%cpu | head -20 | grep -E "(bitcoin|mining|cryptonight|xmrig)") if [ ! -z "$SUSPICIOUS" ]; then log_message "SUSPICIOUS PROCESSES DETECTED" echo "Suspicious processes found: $SUSPICIOUS" | mail -s "Process Alert" $ALERT_EMAIL fi } Main execution check_cpu check_memory check_disk check_processes log_message "System monitoring check completed" ``` Automated Detection Scripts Comprehensive Intrusion Detection Script Create an all-in-one detection script: ```bash #!/bin/bash comprehensive_ids.sh SCRIPT_DIR="/opt/ids" LOG_DIR="/var/log/ids" CONFIG_FILE="$SCRIPT_DIR/ids.conf" Create directories mkdir -p $SCRIPT_DIR $LOG_DIR Configuration ALERT_EMAIL="admin@yourdomain.com" FAILED_LOGIN_THRESHOLD=5 SCAN_THRESHOLD=20 CHECK_INTERVAL=300 # 5 minutes Function to send alerts send_alert() { local subject="$1" local message="$2" echo "$message" | mail -s "$subject" $ALERT_EMAIL logger -p auth.warning "IDS Alert: $subject - $message" } Check for brute force attacks check_brute_force() { local log_file="/var/log/auth.log" local temp_file="/tmp/failed_logins.tmp" # Extract failed login attempts from last 10 minutes grep "$(date --date='10 minutes ago' '+%b %d %H:%M').*Failed password" $log_file 2>/dev/null | \ awk '{print $(NF-3)}' | sort | uniq -c | sort -nr > $temp_file while read count ip; do if [ "$count" -gt "$FAILED_LOGIN_THRESHOLD" ]; then send_alert "Brute Force Attack Detected" "IP $ip has $count failed login attempts" # Optionally block the IP iptables -A INPUT -s $ip -j DROP 2>/dev/null echo "$ip blocked due to brute force attack" >> $LOG_DIR/blocked_ips.log fi done < $temp_file rm -f $temp_file } Check for port scans check_port_scans() { local connections_file="/tmp/current_connections.tmp" netstat -an | grep SYN_RECV | awk '{print $5}' | cut -d: -f1 | \ sort | uniq -c | sort -nr > $connections_file while read count ip; do if [ "$count" -gt "$SCAN_THRESHOLD" ]; then send_alert "Port Scan Detected" "IP $ip has $count SYN_RECV connections" fi done < $connections_file rm -f $connections_file } Check for rootkit indicators check_rootkits() { # Check for common rootkit files local rootkit_files=("/tmp/.ICE-unix/.x" "/tmp/.X11-unix/.x" "/dev/shm/.ice" "/dev/shm/.X11") for file in "${rootkit_files[@]}"; do if [ -f "$file" ]; then send_alert "Potential Rootkit Detected" "Suspicious file found: $file" fi done # Check for suspicious processes if pgrep -f "(bitcoin|mining|cryptonight)" > /dev/null; then send_alert "Cryptocurrency Mining Detected" "Potential cryptomining process detected" fi } Check system integrity check_integrity() { # Check for modified system binaries local critical_bins=("/bin/ls" "/bin/ps" "/bin/netstat" "/usr/bin/who") for bin in "${critical_bins[@]}"; do if [ -f "$bin" ]; then local current_hash=$(sha256sum "$bin" | cut -d' ' -f1) local baseline_file="$LOG_DIR/$(basename $bin).hash" if [ -f "$baseline_file" ]; then local baseline_hash=$(cat "$baseline_file") if [ "$current_hash" != "$baseline_hash" ]; then send_alert "System Binary Modified" "Binary $bin has been modified" fi else echo "$current_hash" > "$baseline_file" fi fi done } Main execution loop main() { echo "$(date): Starting comprehensive IDS check" >> $LOG_DIR/ids.log check_brute_force check_port_scans check_rootkits check_integrity echo "$(date): IDS check completed" >> $LOG_DIR/ids.log } Run the main function main Schedule with cron: /5 * /opt/ids/comprehensive_ids.sh ``` Common Attack Patterns Recognizing Attack Signatures Learn to identify common attack patterns in logs: SQL Injection Attempts ```bash Monitor web logs for SQL injection patterns sudo grep -i "union.select\|drop.table\|insert.into\|update.set" /var/log/apache2/access.log sudo grep -i "1=1\|'.or.'1'='1" /var/log/apache2/access.log ``` Cross-Site Scripting (XSS) ```bash Look for XSS patterns sudo grep -i "> $ALERT_LOG awk '{print $9}' $WEB_LOG | sort | uniq -c | sort -nr | head -10 >> $ALERT_LOG # Alert on excessive 404 errors (potential scanning) local error_404_count=$(awk '$9 == "404" {count++} END {print count+0}' $WEB_LOG) if [ $error_404_count -gt 100 ]; then echo "ALERT: High number of 404 errors detected: $error_404_count" >> $ALERT_LOG fi } Function to analyze user agents analyze_user_agents() { echo "=== Suspicious User Agents ===" >> $ALERT_LOG # Common attack tool user agents grep -i "sqlmap\|nikto\|nmap\|masscan\|zap\|burp\|curl.*bot" $WEB_LOG >> $ALERT_LOG # Empty or suspicious user agents awk -F'"' '$6 == "" || $6 == "-" {print $1 " " $6}' $WEB_LOG | head -20 >> $ALERT_LOG } Function to detect anomalous request patterns detect_anomalies() { echo "=== Request Pattern Anomalies ===" >> $ALERT_LOG # Requests with unusual parameters grep -E "\.php\?.=.http://\|\.php\?.=.ftp://\|\.php\?.=.\.\./\.\." $WEB_LOG >> $ALERT_LOG # Large POST requests (potential file uploads) awk '$6 == "POST" && $10 > 1000000 {print $0}' $WEB_LOG >> $ALERT_LOG } Execute analysis functions echo "$(date): Starting advanced log analysis" >> $ALERT_LOG analyze_status_codes analyze_user_agents detect_anomalies echo "$(date): Advanced log analysis completed" >> $ALERT_LOG ``` Troubleshooting and Common Issues Log Rotation and Storage Issues Manage log files effectively to prevent storage problems: ```bash Configure logrotate for security logs sudo nano /etc/logrotate.d/security-logs Add configuration: /var/log/security_alerts.log { daily rotate 30 compress delaycompress missingok notifempty create 644 root root } Test logrotate configuration sudo logrotate -d /etc/logrotate.d/security-logs ``` Performance Impact Mitigation Optimize monitoring to minimize system impact: ```bash Use ionice for I/O intensive operations ionice -c 3 -p $$ # Add to beginning of scripts Limit CPU usage with cpulimit cpulimit -l 20 -p $$ & # Limit to 20% CPU Use nice for process priority nice -n 19 ./monitoring_script.sh ``` False Positive Reduction Minimize false alerts by tuning detection parameters: ```bash Create whitelist for known good IPs WHITELIST=("192.168.1.100" "10.0.0.50" "203.0.113.10") is_whitelisted() { local ip="$1" for white_ip in "${WHITELIST[@]}"; do if [ "$ip" = "$white_ip" ]; then return 0 fi done return 1 } Use in detection scripts if ! is_whitelisted "$suspicious_ip"; then send_alert "Suspicious Activity" "IP $suspicious_ip detected" fi ``` Debugging Detection Scripts Add debugging capabilities to your scripts: ```bash #!/bin/bash Enable debugging set -x # Print commands as they execute set -e # Exit on error Debug function debug() { if [ "$DEBUG" = "1" ]; then echo "DEBUG: $1" >&2 fi } Usage: DEBUG=1 ./script.sh debug "Starting intrusion detection check" ``` Best Practices Security Hardening Implement security best practices alongside intrusion detection: ```bash Secure log files sudo chmod 640 /var/log/auth.log sudo chown root:adm /var/log/auth.log Implement log forwarding to central server echo ". @@logserver.domain.com:514" >> /etc/rsyslog.conf Enable process accounting sudo apt install acct sudo accton /var/log/account/pacct ``` Automated Response Actions Implement automated responses to detected threats: ```bash #!/bin/bash automated_response.sh block_ip() { local ip="$1" local reason="$2" # Block with iptables iptables -A INPUT -s $ip -j DROP # Log the action echo "$(date): Blocked $ip - Reason: $reason" >> /var/log/blocked_ips.log # Add to fail2ban if available if command -v fail2ban-client &> /dev/null; then fail2ban-client set sshd banip $ip fi } quarantine_file() { local file="$1" local quarantine_dir="/var/quarantine" mkdir -p $quarantine_dir mv "$file" "$quarantine_dir/$(basename $file).$(date +%s)" echo "$(date): Quarantined $file" >> /var/log/quarantine.log } ``` Compliance and Documentation Maintain proper documentation for compliance: ```bash Create incident response template cat > /etc/security/incident_template.txt << EOF SECURITY INCIDENT REPORT Date/Time: Detected By: Incident Type: Affected Systems: Description: Initial Response: Root Cause: Resolution: Lessons Learned: EOF ``` Regular Maintenance Tasks Implement regular maintenance procedures: ```bash #!/bin/bash maintenance.sh Update IDS signatures sudo suricata-update sudo systemctl restart suricata Rotate and compress old logs sudo logrotate -f /etc/logrotate.conf Update AIDE database after system updates sudo aide --update sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db Clean up temporary files find /tmp -name "*.tmp" -mtime +7 -delete Review and archive old alerts find /var/log -name "alert" -mtime +30 -exec gzip {} \; ``` Monitoring the Monitors Ensure your intrusion detection systems are working properly: ```bash #!/bin/bash monitor_health.sh check_service() { local service="$1" if ! systemctl is-active --quiet $service; then echo "ALERT: $service is not running" | mail -s "IDS Service Down" admin@domain.com fi } check_log_growth() { local logfile="$1" local size_threshold="$2" if [ -f "$logfile" ]; then local size=$(stat -c%s "$logfile") if [ $size -gt $size_threshold ]; then echo "ALERT: $logfile is growing too large ($size bytes)" | mail -s "Log Size Alert" admin@domain.com fi fi } Check critical services check_service "suricata" check_service "fail2ban" check_service "rsyslog" Check log file sizes check_log_growth "/var/log/suricata/eve.json" 1073741824 # 1GB check_log_growth "/var/log/auth.log" 536870912 # 512MB ``` Conclusion Implementing effective intrusion detection on Linux systems requires a comprehensive approach combining multiple detection methods, automated responses, and continuous monitoring. The strategies and tools covered in this guide provide a solid foundation for detecting and responding to security threats. Key Takeaways 1. Multi-layered Approach: Combine network-based and host-based detection methods for comprehensive coverage 2. Log Analysis: Regular analysis of system logs is fundamental to intrusion detection 3. Automation: Automated monitoring and response capabilities improve detection speed and reduce manual effort 4. False Positive Management: Proper tuning and whitelisting reduce alert fatigue 5. Regular Maintenance: Keep detection systems updated and properly maintained Next Steps To further enhance your Linux intrusion detection capabilities: 1. Implement centralized logging with tools like ELK Stack or Splunk 2. Integrate with Security Information and Event Management (SIEM) systems 3. Develop custom detection rules based on your specific environment 4. Establish incident response procedures and team training 5. Regular security assessments and penetration testing to validate detection effectiveness Remember that intrusion detection is an ongoing process that requires continuous refinement and adaptation to emerging threats. Stay informed about new attack vectors and update your detection mechanisms accordingly. By following the practices outlined in this guide, you'll significantly improve your ability to detect and respond to intrusions on Linux systems, helping to maintain the security and integrity