How to configure rsyslog in Linux

How to Configure rsyslog in Linux System logging is a critical component of Linux administration that enables administrators to monitor system activities, troubleshoot issues, and maintain security. rsyslog (reliable syslog) is the default logging daemon in most modern Linux distributions, providing enhanced functionality over traditional syslog implementations. This comprehensive guide will walk you through configuring rsyslog from basic setup to advanced configurations, ensuring you can effectively manage your system's logging infrastructure. Table of Contents 1. [Introduction to rsyslog](#introduction-to-rsyslog) 2. [Prerequisites and Requirements](#prerequisites-and-requirements) 3. [Understanding rsyslog Architecture](#understanding-rsyslog-architecture) 4. [Basic Configuration](#basic-configuration) 5. [Advanced Configuration Options](#advanced-configuration-options) 6. [Practical Examples and Use Cases](#practical-examples-and-use-cases) 7. [Remote Logging Configuration](#remote-logging-configuration) 8. [Log Rotation and Management](#log-rotation-and-management) 9. [Security Considerations](#security-considerations) 10. [Troubleshooting Common Issues](#troubleshooting-common-issues) 11. [Best Practices](#best-practices) 12. [Conclusion](#conclusion) Introduction to rsyslog rsyslog is a powerful, high-performance logging system that extends the basic syslog protocol with advanced features such as reliable delivery, encryption, database storage, and flexible configuration options. Unlike traditional syslog, rsyslog offers multi-threading capabilities, TCP support, and various output formats, making it suitable for enterprise environments and complex logging requirements. Key features of rsyslog include: - High-performance logging with multi-threading support - Reliable message delivery with queue management - Support for multiple input and output modules - Advanced filtering capabilities - Database integration (MySQL, PostgreSQL, Oracle) - Encryption and authentication for secure remote logging - Custom message formats and templates Prerequisites and Requirements Before configuring rsyslog, ensure you have the following prerequisites: System Requirements - Linux distribution with rsyslog installed (Ubuntu, CentOS, RHEL, Debian, SUSE) - Root or sudo access to the system - Basic understanding of Linux command line - Text editor (vi, nano, or your preferred editor) Software Installation Most modern Linux distributions include rsyslog by default. To verify installation or install rsyslog: Ubuntu/Debian: ```bash sudo apt update sudo apt install rsyslog ``` CentOS/RHEL/Fedora: ```bash sudo yum install rsyslog or for newer versions sudo dnf install rsyslog ``` Check rsyslog status: ```bash sudo systemctl status rsyslog sudo systemctl enable rsyslog sudo systemctl start rsyslog ``` Understanding rsyslog Architecture rsyslog follows a modular architecture with three main components: Input Modules Input modules collect log messages from various sources: - `imuxsock`: Unix socket input (default for local logging) - `imklog`: Kernel logging - `imfile`: File monitoring - `imtcp`: TCP input for remote logging - `imudp`: UDP input for remote logging Processing Engine The processing engine applies rules, filters, and transformations to log messages based on: - Facility: Type of program generating the message - Severity: Importance level of the message - Custom filters and conditions Output Modules Output modules determine where log messages are stored or forwarded: - `omfile`: File output (default) - `omfwd`: Forward to remote servers - `ommysql`: MySQL database output - `ommail`: Email notifications - `omstdout`: Standard output Basic Configuration The main rsyslog configuration file is typically located at `/etc/rsyslog.conf`, with additional configuration files in `/etc/rsyslog.d/`. Let's examine the basic configuration structure. Configuration File Structure ```bash sudo nano /etc/rsyslog.conf ``` A typical rsyslog configuration includes: ``` Modules $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support Global directives $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $RepeatedMsgReduction on $FileOwner syslog $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 Rules *.info;mail.none;authpriv.none;cron.none /var/log/messages authpriv.* /var/log/secure mail.* /var/log/maillog cron.* /var/log/cron .emerg :omusrmsg: uucp,news.crit /var/log/spooler local7.* /var/log/boot.log ``` Understanding Selectors and Actions rsyslog rules consist of selectors and actions: Selector Format: `facility.priority` - Facility: Type of program (mail, kern, user, daemon, etc.) - Priority: Message severity (debug, info, notice, warning, err, crit, alert, emerg) Common Facilities: - `kern`: Kernel messages - `user`: User-level messages - `mail`: Mail system messages - `daemon`: System daemon messages - `auth`: Security/authorization messages - `syslog`: Messages generated by syslogd - `local0-local7`: Custom facilities Priority Levels (lowest to highest): - `debug`: Debug-level messages - `info`: Informational messages - `notice`: Normal but significant conditions - `warning`: Warning conditions - `err`: Error conditions - `crit`: Critical conditions - `alert`: Action must be taken immediately - `emerg`: System is unusable Basic Configuration Examples Log all messages except mail to /var/log/messages: ``` *.info;mail.none /var/log/messages ``` Log authentication messages to a separate file: ``` auth.* /var/log/auth.log ``` Log kernel messages with priority warning and above: ``` kern.warning /var/log/kernel-warnings.log ``` Advanced Configuration Options Custom Templates Templates define the format of log messages. Create custom templates for specific requirements: ``` Define custom template $template CustomFormat,"%timegenerated% %HOSTNAME% %syslogtag% %msg%\n" Use template for specific logs local0.* /var/log/custom.log;CustomFormat ``` Template with JSON format: ``` $template JsonFormat,"{ \"timestamp\":\"%timegenerated:::date-rfc3339%\", \"host\":\"%HOSTNAME%\", \"severity\":\"%syslogpriority-text%\", \"facility\":\"%syslogfacility-text%\", \"tag\":\"%syslogtag%\", \"message\":\"%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\" }\n" local1.* /var/log/app.json;JsonFormat ``` Property-Based Filters Use property-based filters for advanced message filtering: ``` Filter by hostname :hostname, isequal, "webserver01" /var/log/webserver01.log Filter by program name :programname, isequal, "sshd" /var/log/ssh.log Filter using regular expressions :msg, regex, "Failed password.*" /var/log/failed-logins.log Filter by IP address :msg, contains, "192.168.1." /var/log/local-network.log ``` Dynamic File Names Create dynamic file names based on message properties: ``` $template DynamicFile,"/var/log/%HOSTNAME%/%programname%.log" . ?DynamicFile ``` Conditional Processing Implement conditional logic in rsyslog configuration: ``` Stop processing if message matches condition :programname, isequal, "spam-program" stop Process only during specific hours $template HourlyLog,"/var/log/hourly/%$HOUR%.log" if $hour >= 9 and $hour <= 17 then { *.info ?HourlyLog } ``` Practical Examples and Use Cases Example 1: Web Server Logging Configure rsyslog for comprehensive web server logging: ``` /etc/rsyslog.d/50-webserver.conf Apache access logs $InputFileName /var/log/apache2/access.log $InputFileTag apache-access: $InputFileStateFile apache-access-state $InputFileSeverity info $InputFileFacility local0 $InputRunFileMonitor Apache error logs $InputFileName /var/log/apache2/error.log $InputFileTag apache-error: $InputFileStateFile apache-error-state $InputFileSeverity error $InputFileFacility local1 $InputRunFileMonitor Separate processing local0.* /var/log/webserver/access.log local1.* /var/log/webserver/error.log Stop further processing local0.* stop local1.* stop ``` Example 2: Database Server Monitoring Monitor database activities with custom logging: ``` /etc/rsyslog.d/60-database.conf MySQL slow queries $InputFileName /var/log/mysql/slow.log $InputFileTag mysql-slow: $InputFileStateFile mysql-slow-state $InputFileSeverity warning $InputFileFacility local2 $InputRunFileMonitor Custom template for database logs $template DatabaseFormat,"%timegenerated% [%syslogfacility-text%] %hostname% %msg%\n" Database logging rules local2.* /var/log/database/slow-queries.log;DatabaseFormat Email alerts for critical database issues local2.crit :ommail:admin@company.com ``` Example 3: Security Event Logging Implement comprehensive security logging: ``` /etc/rsyslog.d/70-security.conf Security template $template SecurityFormat,"%timegenerated:::date-rfc3339% %HOSTNAME% %syslogtag% %msg%\n" Authentication events auth,authpriv.* /var/log/security/auth.log;SecurityFormat Failed login attempts :msg, regex, "(Failed|failed|FAILED).*" /var/log/security/failed-attempts.log;SecurityFormat Sudo usage :programname, isequal, "sudo" /var/log/security/sudo.log;SecurityFormat SSH connections :programname, isequal, "sshd" /var/log/security/ssh.log;SecurityFormat Stop processing for security events auth,authpriv.* stop ``` Remote Logging Configuration Configuring rsyslog Server (Log Receiver) Set up a central logging server to receive logs from multiple clients: ```bash Edit /etc/rsyslog.conf Enable UDP reception (port 514) $ModLoad imudp $UDPServerRun 514 $UDPServerAddress 0.0.0.0 Enable TCP reception (port 514) $ModLoad imtcp $InputTCPServerRun 514 Template for remote logs $template RemoteStore, "/var/log/remote/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%.log" Store remote logs separately if $fromhost-ip != '127.0.0.1' then ?RemoteStore & stop ``` Configuring rsyslog Client (Log Sender) Configure clients to send logs to the central server: ```bash Add to /etc/rsyslog.conf or /etc/rsyslog.d/49-remote.conf Send all logs to remote server via UDP . @192.168.1.100:514 Send all logs to remote server via TCP (more reliable) . @@192.168.1.100:514 Send only specific logs to remote server mail.* @@logserver.company.com:514 kern.* @@logserver.company.com:514 ``` Secure Remote Logging with TLS Configure encrypted remote logging: Server Configuration: ```bash /etc/rsyslog.d/tls.conf Load required modules $ModLoad imtcp TLS configuration $DefaultNetstreamDriver gtls $DefaultNetstreamDriverCAFile /etc/ssl/rsyslog/ca.pem $DefaultNetstreamDriverCertFile /etc/ssl/rsyslog/server-cert.pem $DefaultNetstreamDriverKeyFile /etc/ssl/rsyslog/server-key.pem $InputTCPServerStreamDriverAuthMode x509/name $InputTCPServerStreamDriverPermittedPeer *.company.com $InputTCPServerStreamDriverMode 1 $InputTCPServerRun 6514 ``` Client Configuration: ```bash TLS client configuration $DefaultNetstreamDriverCAFile /etc/ssl/rsyslog/ca.pem $ActionSendStreamDriver gtls $ActionSendStreamDriverMode 1 $ActionSendStreamDriverAuthMode x509/name $ActionSendStreamDriverPermittedPeer logserver.company.com Send logs via TLS . @@logserver.company.com:6514 ``` Log Rotation and Management Configuring logrotate Manage log file rotation to prevent disk space issues: ```bash /etc/logrotate.d/rsyslog /var/log/syslog /var/log/mail.info /var/log/mail.warn /var/log/mail.err /var/log/mail.log /var/log/daemon.log /var/log/kern.log /var/log/auth.log /var/log/user.log /var/log/lpr.log /var/log/cron.log /var/log/debug /var/log/messages { rotate 7 daily missingok notifempty delaycompress compress postrotate /usr/lib/rsyslog/rsyslog-rotate endscript } ``` Custom Log Rotation Create custom rotation for application logs: ```bash /etc/logrotate.d/custom-app /var/log/custom/*.log { daily rotate 30 compress delaycompress missingok notifempty create 0644 syslog adm postrotate systemctl reload rsyslog endscript } ``` Security Considerations File Permissions and Ownership Secure log files with appropriate permissions: ```bash Set secure permissions sudo chmod 640 /var/log/*.log sudo chown syslog:adm /var/log/*.log Configure in rsyslog.conf $FileOwner syslog $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 ``` Rate Limiting Prevent log flooding attacks: ```bash Rate limiting configuration $SystemLogRateLimitInterval 2 $SystemLogRateLimitBurst 50 Per-process rate limiting $imjournalRatelimitInterval 600 $imjournalRatelimitBurst 20000 ``` Input Validation Implement input validation for remote logs: ```bash Reject oversized messages $MaxMessageSize 4k Validate hostnames $InputTCPServerStreamDriverPermittedPeer *.trusted-domain.com ``` Troubleshooting Common Issues Issue 1: rsyslog Not Starting Symptoms: Service fails to start or stops unexpectedly Diagnosis: ```bash sudo systemctl status rsyslog sudo journalctl -u rsyslog sudo rsyslogd -N1 # Test configuration syntax ``` Common Solutions: - Check configuration syntax errors - Verify file permissions - Ensure required modules are available - Check disk space in log directories Issue 2: Logs Not Being Written Symptoms: Expected log messages not appearing in files Diagnosis: ```bash Test logging logger "Test message" tail -f /var/log/syslog Check rsyslog stats sudo kill -USR1 $(cat /var/run/rsyslogd.pid) tail /var/log/messages ``` Solutions: - Verify selector syntax in configuration - Check file permissions and ownership - Ensure target directories exist - Review filter rules for conflicts Issue 3: Remote Logging Not Working Symptoms: Remote logs not received by server Diagnosis: ```bash Check network connectivity telnet logserver.company.com 514 Monitor network traffic sudo tcpdump -i any port 514 Test with logger logger -n logserver.company.com "Test remote message" ``` Solutions: - Verify firewall rules allow syslog traffic - Check server listening configuration - Validate client forwarding rules - Test with both UDP and TCP Issue 4: High CPU Usage Symptoms: rsyslog consuming excessive CPU resources Diagnosis: ```bash Monitor rsyslog processes top -p $(pgrep rsyslog) Check for log loops sudo lsof +D /var/log/ ``` Solutions: - Review filter rules for efficiency - Implement rate limiting - Check for logging loops - Optimize template usage Issue 5: Disk Space Issues Symptoms: Log files consuming excessive disk space Solutions: ```bash Implement log rotation sudo logrotate -f /etc/logrotate.d/rsyslog Monitor log sizes du -sh /var/log/* Configure size-based rotation $outchannel logfile,/var/log/file.log,52428800,/bin/rm /var/log/file.log . :omfile:$logfile ``` Best Practices Configuration Management 1. Use Modular Configuration: - Split configurations into separate files in `/etc/rsyslog.d/` - Use descriptive filenames with numeric prefixes for ordering - Document configuration changes 2. Version Control: - Keep configuration files in version control - Test changes in development environment first - Maintain configuration documentation Performance Optimization 1. Efficient Filtering: - Place most specific filters first - Use property-based filters for complex conditions - Implement stop rules to prevent unnecessary processing 2. Queue Management: ```bash # Configure work queues for high-volume logging $MainMsgQueueSize 100000 $MainMsgQueueHighWaterMark 80000 $MainMsgQueueLowWaterMark 20000 $MainMsgQueueDiscardMark 98000 ``` 3. Asynchronous Processing: ```bash # Enable asynchronous processing $OMFileAsyncWriting on $OMFileFlushInterval 1 ``` Security Best Practices 1. Principle of Least Privilege: - Run rsyslog with minimal required permissions - Restrict access to log files - Use dedicated log user accounts 2. Network Security: - Use TLS for remote logging - Implement firewall rules - Validate remote log sources 3. Log Integrity: - Implement log signing for critical logs - Use immutable storage when possible - Regular log integrity checks Monitoring and Maintenance 1. Regular Monitoring: - Monitor rsyslog service status - Check log file sizes and rotation - Verify remote logging connectivity 2. Automated Alerting: ```bash # Alert on critical messages *.crit :ommail:admin@company.com # Monitor rsyslog health if $programname == 'rsyslogd' and $msg contains 'error' then { :ommail:sysadmin@company.com } ``` 3. Performance Monitoring: - Track message processing rates - Monitor queue sizes - Check system resource usage Conclusion rsyslog is a powerful and flexible logging solution that can be configured to meet diverse requirements from simple local logging to complex enterprise-wide log management systems. This comprehensive guide has covered the essential aspects of rsyslog configuration, from basic setup to advanced features like remote logging, security configurations, and performance optimization. Key takeaways from this guide: 1. Start Simple: Begin with basic configurations and gradually implement advanced features as needed 2. Plan Your Architecture: Design your logging infrastructure considering scalability, security, and maintenance requirements 3. Test Thoroughly: Always test configuration changes in a development environment before deploying to production 4. Monitor Continuously: Implement monitoring and alerting to ensure your logging system remains healthy and effective 5. Document Everything: Maintain comprehensive documentation of your rsyslog configurations and procedures By following the practices and examples outlined in this guide, you'll be well-equipped to implement and maintain a robust logging infrastructure that provides valuable insights into your Linux systems while maintaining security and performance standards. Remember that effective logging is not just about collecting data—it's about collecting the right data in the right format and making it accessible for analysis and troubleshooting. Regular review and optimization of your rsyslog configuration will ensure it continues to meet your evolving needs and provides maximum value for system administration and security monitoring. For continued learning, consider exploring advanced rsyslog modules, integration with log analysis tools like ELK stack (Elasticsearch, Logstash, Kibana), and implementing automated log analysis and alerting systems to further enhance your logging infrastructure capabilities.