How to rotate logs in Linux
How to Rotate Logs in Linux
Log rotation is a critical system administration task that prevents log files from consuming excessive disk space while maintaining historical records for troubleshooting and compliance purposes. This comprehensive guide covers everything you need to know about rotating logs in Linux systems, from basic concepts to advanced configurations.
Table of Contents
1. [Introduction to Log Rotation](#introduction-to-log-rotation)
2. [Prerequisites and Requirements](#prerequisites-and-requirements)
3. [Understanding Logrotate](#understanding-logrotate)
4. [Basic Logrotate Configuration](#basic-logrotate-configuration)
5. [Advanced Configuration Options](#advanced-configuration-options)
6. [Manual Log Rotation Methods](#manual-log-rotation-methods)
7. [Systemd Journal Log Rotation](#systemd-journal-log-rotation)
8. [Application-Specific Log Rotation](#application-specific-log-rotation)
9. [Troubleshooting Common Issues](#troubleshooting-common-issues)
10. [Best Practices and Tips](#best-practices-and-tips)
11. [Monitoring and Maintenance](#monitoring-and-maintenance)
12. [Conclusion](#conclusion)
Introduction to Log Rotation
Log rotation is the automated process of archiving current log files and creating new ones to prevent unlimited growth of log data. Without proper log rotation, log files can grow to consume all available disk space, potentially causing system failures and performance issues.
Why Log Rotation Matters
- Disk Space Management: Prevents logs from filling up disk partitions
- Performance Optimization: Smaller log files are faster to search and process
- Compliance Requirements: Maintains historical data for auditing purposes
- System Stability: Prevents disk space exhaustion that could crash services
- Backup Efficiency: Smaller, organized log files are easier to backup
Common Log Rotation Strategies
1. Size-based rotation: Rotate when files reach a specific size
2. Time-based rotation: Rotate at regular intervals (daily, weekly, monthly)
3. Hybrid approach: Combine size and time-based criteria
4. Manual rotation: Rotate logs on-demand or during maintenance windows
Prerequisites and Requirements
Before implementing log rotation, ensure you have:
System Requirements
- Linux system with root or sudo privileges
- Basic understanding of file permissions and ownership
- Familiarity with cron jobs and system services
- Knowledge of text editors (vi, nano, or similar)
Essential Tools
Most Linux distributions include these tools by default:
```bash
Check if logrotate is installed
which logrotate
logrotate --version
Install logrotate if missing (Ubuntu/Debian)
sudo apt update && sudo apt install logrotate
Install logrotate if missing (CentOS/RHEL/Fedora)
sudo yum install logrotate
or for newer versions
sudo dnf install logrotate
```
Understanding Log File Locations
Common log file locations in Linux:
- `/var/log/syslog` - System messages
- `/var/log/auth.log` - Authentication logs
- `/var/log/kern.log` - Kernel messages
- `/var/log/apache2/` - Apache web server logs
- `/var/log/nginx/` - Nginx web server logs
- `/var/log/mysql/` - MySQL database logs
Understanding Logrotate
Logrotate is the standard log rotation utility in most Linux distributions. It reads configuration files to determine how to handle different log files.
How Logrotate Works
1. Configuration Reading: Reads `/etc/logrotate.conf` and files in `/etc/logrotate.d/`
2. Condition Checking: Evaluates rotation criteria (size, time, etc.)
3. File Processing: Rotates files that meet the criteria
4. Post-processing: Executes scripts and sends signals to services
5. State Management: Updates state file to track rotation history
Logrotate File Structure
```bash
Main configuration file
/etc/logrotate.conf
Service-specific configurations
/etc/logrotate.d/
├── apache2
├── nginx
├── mysql-server
├── rsyslog
└── custom-app
State file (tracks last rotation)
/var/lib/logrotate/status
```
Basic Logrotate Syntax
```bash
Basic structure
/path/to/logfile {
rotation_criteria
number_of_files_to_keep
additional_options
pre/post_rotation_scripts
}
```
Basic Logrotate Configuration
Global Configuration
The main configuration file `/etc/logrotate.conf` contains global settings:
```bash
Sample /etc/logrotate.conf
Rotate logs weekly by default
weekly
Keep 4 weeks worth of backlogs
rotate 4
Create new (empty) log files after rotating old ones
create
Use date as a suffix of the rotated file
dateext
Compress rotated files
compress
Include all files in /etc/logrotate.d/
include /etc/logrotate.d
Specific configuration for wtmp
/var/log/wtmp {
monthly
create 0664 root utmp
minsize 1M
rotate 1
}
```
Creating a Basic Configuration
Create a custom logrotate configuration for your application:
```bash
sudo nano /etc/logrotate.d/myapp
```
Add the following configuration:
```bash
/var/log/myapp/*.log {
daily
rotate 30
compress
delaycompress
missingok
notifempty
create 0644 myapp myapp
postrotate
systemctl reload myapp > /dev/null 2>&1 || true
endscript
}
```
Configuration Options Explained
| Option | Description |
|--------|-------------|
| `daily/weekly/monthly` | Rotation frequency |
| `rotate N` | Keep N rotated files |
| `compress` | Compress rotated files with gzip |
| `delaycompress` | Compress previous rotation, not current |
| `missingok` | Don't error if log file is missing |
| `notifempty` | Don't rotate empty files |
| `create mode owner group` | Create new log file with specified permissions |
| `copytruncate` | Copy file contents then truncate original |
| `dateext` | Use date extension instead of numbers |
Advanced Configuration Options
Size-Based Rotation
Rotate logs when they reach a specific size:
```bash
/var/log/large-app.log {
size 100M
rotate 5
compress
notifempty
create 0644 app-user app-group
postrotate
/usr/bin/killall -HUP large-app
endscript
}
```
Multiple Log Files
Handle multiple log files with different patterns:
```bash
Multiple specific files
/var/log/app1.log /var/log/app2.log {
weekly
rotate 12
compress
missingok
notifempty
}
Wildcard patterns
/var/log/cluster/*.log {
daily
rotate 7
compress
sharedscripts
postrotate
systemctl reload cluster-service
endscript
}
```
Advanced Compression Options
```bash
/var/log/database/*.log {
daily
rotate 90
compress
compresscmd /bin/bzip2
compressext .bz2
compressoptions -9
delaycompress
notifempty
}
```
Conditional Rotation
Use scripts to determine rotation conditions:
```bash
/var/log/conditional.log {
daily
rotate 30
missingok
prerotate
if [ -d /var/run/myservice ]; then
echo "Service is running, proceeding with rotation"
else
echo "Service not running, skipping rotation"
exit 1
fi
endscript
postrotate
systemctl reload myservice
endscript
}
```
Manual Log Rotation Methods
Simple Manual Rotation
For one-time or custom rotation needs:
```bash
#!/bin/bash
Simple log rotation script
LOGFILE="/var/log/myapp.log"
BACKUP_DIR="/var/log/archive"
DATE=$(date +%Y%m%d_%H%M%S)
Create backup directory if it doesn't exist
mkdir -p "$BACKUP_DIR"
Check if log file exists and is not empty
if [ -s "$LOGFILE" ]; then
# Copy and compress the log file
cp "$LOGFILE" "$BACKUP_DIR/myapp_${DATE}.log"
gzip "$BACKUP_DIR/myapp_${DATE}.log"
# Truncate the original log file
> "$LOGFILE"
# Restart the service to recognize the new log file
systemctl reload myapp
echo "Log rotated successfully: myapp_${DATE}.log.gz"
else
echo "Log file is empty or doesn't exist"
fi
```
Advanced Manual Rotation Script
```bash
#!/bin/bash
Advanced log rotation script with retention policy
LOGFILE="/var/log/myapp.log"
BACKUP_DIR="/var/log/archive"
RETENTION_DAYS=30
MAX_SIZE="100M"
Function to convert size to bytes
size_to_bytes() {
local size=$1
local number=${size%[A-Za-z]*}
local unit=${size#$number}
case $unit in
K|k) echo $((number * 1024)) ;;
M|m) echo $((number 1024 1024)) ;;
G|g) echo $((number 1024 1024 * 1024)) ;;
*) echo $number ;;
esac
}
Check if rotation is needed
if [ -f "$LOGFILE" ]; then
CURRENT_SIZE=$(stat -f%z "$LOGFILE" 2>/dev/null || stat -c%s "$LOGFILE")
MAX_BYTES=$(size_to_bytes "$MAX_SIZE")
if [ "$CURRENT_SIZE" -gt "$MAX_BYTES" ]; then
echo "Log file size ($CURRENT_SIZE bytes) exceeds maximum ($MAX_BYTES bytes)"
# Perform rotation
DATE=$(date +%Y%m%d_%H%M%S)
mkdir -p "$BACKUP_DIR"
mv "$LOGFILE" "$BACKUP_DIR/myapp_${DATE}.log"
gzip "$BACKUP_DIR/myapp_${DATE}.log"
# Create new log file with proper permissions
touch "$LOGFILE"
chown myapp:myapp "$LOGFILE"
chmod 644 "$LOGFILE"
# Signal application to reopen log file
pkill -HUP myapp
echo "Log rotated: myapp_${DATE}.log.gz"
else
echo "Log file size is within limits"
fi
fi
Clean up old backups
find "$BACKUP_DIR" -name "myapp_*.log.gz" -mtime +$RETENTION_DAYS -delete
echo "Cleaned up backups older than $RETENTION_DAYS days"
```
Integrating Manual Scripts with Cron
Add your rotation script to cron for automated execution:
```bash
Edit crontab
crontab -e
Add entry to run daily at 2 AM
0 2 * /usr/local/bin/rotate-logs.sh >> /var/log/rotation.log 2>&1
Run weekly on Sunday at 3 AM
0 3 0 /usr/local/bin/weekly-rotation.sh
Run when disk usage exceeds threshold
/15 * [ $(df /var/log | awk 'NR==2 {print $5}' | sed 's/%//') -gt 80 ] && /usr/local/bin/emergency-rotation.sh
```
Systemd Journal Log Rotation
Modern Linux systems using systemd have built-in journal log management.
Configuring Journald
Edit the journald configuration:
```bash
sudo nano /etc/systemd/journald.conf
```
Key configuration options:
```bash
[Journal]
Limit journal size
SystemMaxUse=1G
SystemKeepFree=500M
SystemMaxFileSize=100M
Retention settings
MaxRetentionSec=1month
MaxFileSec=1week
Forward to syslog
ForwardToSyslog=yes
ForwardToWall=no
Storage location
Storage=persistent
```
Manual Journal Cleanup
```bash
View current journal disk usage
journalctl --disk-usage
Clean journals older than 2 weeks
sudo journalctl --vacuum-time=2weeks
Limit journal size to 500MB
sudo journalctl --vacuum-size=500M
Keep only last 10 journal files
sudo journalctl --vacuum-files=10
Verify journal integrity
sudo journalctl --verify
```
Automated Journal Rotation Script
```bash
#!/bin/bash
Systemd journal rotation script
MAX_SIZE="500M"
MAX_TIME="1month"
LOG_FILE="/var/log/journal-rotation.log"
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S'): $1" >> "$LOG_FILE"
}
Check current usage
CURRENT_USAGE=$(journalctl --disk-usage | grep -o '[0-9.]*[KMGT]B')
log_message "Current journal usage: $CURRENT_USAGE"
Perform cleanup
log_message "Starting journal cleanup"
Clean by size
journalctl --vacuum-size="$MAX_SIZE" >> "$LOG_FILE" 2>&1
Clean by time
journalctl --vacuum-time="$MAX_TIME" >> "$LOG_FILE" 2>&1
Verify integrity after cleanup
if journalctl --verify >> "$LOG_FILE" 2>&1; then
log_message "Journal cleanup completed successfully"
else
log_message "Warning: Journal verification failed after cleanup"
fi
Report new usage
NEW_USAGE=$(journalctl --disk-usage | grep -o '[0-9.]*[KMGT]B')
log_message "New journal usage: $NEW_USAGE"
```
Application-Specific Log Rotation
Apache Web Server
```bash
/etc/logrotate.d/apache2
/var/log/apache2/*.log {
daily
missingok
rotate 52
compress
delaycompress
notifempty
create 0640 www-data adm
sharedscripts
postrotate
if /etc/init.d/apache2 status > /dev/null ; then \
/etc/init.d/apache2 reload > /dev/null; \
fi;
endscript
prerotate
if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
run-parts /etc/logrotate.d/httpd-prerotate; \
fi;
endscript
}
```
Nginx Web Server
```bash
/etc/logrotate.d/nginx
/var/log/nginx/*.log {
daily
missingok
rotate 30
compress
delaycompress
notifempty
create 0644 nginx nginx
sharedscripts
prerotate
if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
run-parts /etc/logrotate.d/httpd-prerotate; \
fi
endscript
postrotate
invoke-rc.d nginx rotate >/dev/null 2>&1
endscript
}
```
MySQL Database
```bash
/etc/logrotate.d/mysql-server
/var/log/mysql/*.log {
daily
rotate 30
compress
delaycompress
missingok
notifempty
sharedscripts
copytruncate
postrotate
test -x /usr/bin/mysqladmin && \
/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf \
--local flush-error-log \
flush-engine-log \
flush-general-log \
flush-slow-log
endscript
}
```
Custom Application Configuration
```bash
/etc/logrotate.d/custom-app
/opt/myapp/logs/*.log {
hourly
size 50M
rotate 168 # Keep 1 week of hourly logs
compress
delaycompress
notifempty
missingok
create 0644 myapp myapp
sharedscripts
postrotate
# Send USR1 signal to reload logs
/bin/kill -USR1 $(cat /var/run/myapp.pid) 2>/dev/null || true
endscript
prerotate
# Custom pre-rotation commands
/opt/myapp/bin/prepare-rotation.sh
endscript
}
```
Troubleshooting Common Issues
Testing Logrotate Configuration
Before deploying configurations, always test them:
```bash
Test specific configuration file
sudo logrotate -d /etc/logrotate.d/myapp
Test all configurations
sudo logrotate -d /etc/logrotate.conf
Force rotation for testing (dry run)
sudo logrotate -f -v /etc/logrotate.d/myapp
Actually force rotation
sudo logrotate -f /etc/logrotate.d/myapp
```
Common Error Messages and Solutions
"error: skipping because parent directory has insecure permissions"
Problem: Directory permissions are too permissive.
Solution:
```bash
Fix directory permissions
sudo chmod 755 /var/log/myapp
sudo chown root:root /var/log/myapp
Or allow insecure permissions (not recommended)
Add 'su root root' to configuration
```
"error: destination /var/log/myapp.log.1 already exists, skipping rotation"
Problem: Previous rotation files weren't cleaned up properly.
Solution:
```bash
Remove old rotation files
sudo rm /var/log/myapp.log.1
Or use dateext to avoid conflicts
Add 'dateext' to configuration
```
"error: error running shared postrotate script"
Problem: Post-rotation script is failing.
Solution:
```bash
Test the script manually
sudo /etc/init.d/myservice reload
Add error handling to script
postrotate
/etc/init.d/myservice reload > /dev/null 2>&1 || true
endscript
```
Debugging Logrotate Issues
Enable verbose logging and check system logs:
```bash
Run with verbose output
sudo logrotate -v /etc/logrotate.conf
Check system logs for errors
sudo journalctl -u logrotate
sudo grep logrotate /var/log/syslog
Check logrotate status file
sudo cat /var/lib/logrotate/status
Manually update status for testing
sudo nano /var/lib/logrotate/status
```
Permission Issues
Common permission problems and solutions:
```bash
Fix log file ownership
sudo chown myapp:myapp /var/log/myapp.log
sudo chmod 644 /var/log/myapp.log
Fix directory permissions
sudo chmod 755 /var/log/myapp
sudo chown myapp:myapp /var/log/myapp
SELinux context issues (if applicable)
sudo restorecon -R /var/log/myapp
sudo setsebool -P logrotate_use_nfs on # For NFS mounted logs
```
Best Practices and Tips
Configuration Best Practices
1. Use Service-Specific Configurations: Create separate files in `/etc/logrotate.d/` for each service.
2. Test Before Deployment: Always test configurations with `-d` flag before implementing.
3. Monitor Disk Usage: Implement monitoring to alert on high disk usage.
4. Document Custom Configurations: Add comments explaining non-standard settings.
```bash
Good example with documentation
/var/log/myapp/*.log {
# Rotate daily due to high volume application
daily
# Keep 90 days for compliance requirements
rotate 90
# Compress to save space, delay to avoid I/O spike
compress
delaycompress
# Application handles missing log files gracefully
missingok
# Don't rotate empty files to reduce overhead
notifempty
# Maintain specific ownership for application access
create 0644 myapp myapp
# Reload application to recognize new log file
postrotate
systemctl reload myapp > /dev/null 2>&1 || true
endscript
}
```
Performance Considerations
1. Stagger Rotation Times: Avoid rotating all logs simultaneously.
```bash
Distribute rotation across different times
Web server logs at 2 AM
0 2 * /usr/sbin/logrotate /etc/logrotate.d/apache2
Database logs at 3 AM
0 3 * /usr/sbin/logrotate /etc/logrotate.d/mysql
Application logs at 4 AM
0 4 * /usr/sbin/logrotate /etc/logrotate.d/myapp
```
2. Use Appropriate Compression: Balance compression ratio vs. CPU usage.
```bash
Fast compression for frequent rotation
compress
compresscmd /bin/gzip
compressoptions -1
High compression for archival
compress
compresscmd /bin/bzip2
compressoptions -9
```
Security Considerations
1. Proper File Permissions: Ensure rotated logs maintain appropriate permissions.
2. Secure Backup Locations: Store archived logs in secure locations.
3. Log Integrity: Consider signing or checksumming important logs.
```bash
#!/bin/bash
Secure log rotation with integrity checking
LOGFILE="/var/log/secure.log"
ARCHIVE_DIR="/var/log/archive"
DATE=$(date +%Y%m%d_%H%M%S)
if [ -s "$LOGFILE" ]; then
# Create secure archive directory
mkdir -p "$ARCHIVE_DIR"
chmod 700 "$ARCHIVE_DIR"
# Copy and compress log
cp "$LOGFILE" "$ARCHIVE_DIR/secure_${DATE}.log"
# Create checksum for integrity verification
sha256sum "$ARCHIVE_DIR/secure_${DATE}.log" > "$ARCHIVE_DIR/secure_${DATE}.log.sha256"
# Compress the log file
gzip "$ARCHIVE_DIR/secure_${DATE}.log"
# Secure permissions on archived files
chmod 600 "$ARCHIVE_DIR"/secure_${DATE}.*
# Clear original log
> "$LOGFILE"
echo "Secure log rotation completed: secure_${DATE}.log.gz"
fi
```
Automation and Monitoring
Create monitoring scripts to ensure rotation is working:
```bash
#!/bin/bash
Log rotation monitoring script
ALERT_EMAIL="admin@example.com"
LOG_DIRS=("/var/log/apache2" "/var/log/mysql" "/var/log/myapp")
MAX_SIZE_MB=500
for dir in "${LOG_DIRS[@]}"; do
if [ -d "$dir" ]; then
# Find large log files
large_files=$(find "$dir" -name "*.log" -size +${MAX_SIZE_MB}M)
if [ -n "$large_files" ]; then
echo "Large log files found in $dir:" | mail -s "Log Rotation Alert" "$ALERT_EMAIL"
echo "$large_files" | mail -s "Log Rotation Alert" "$ALERT_EMAIL"
fi
# Check for very old log files
old_files=$(find "$dir" -name "*.log" -mtime +30)
if [ -n "$old_files" ]; then
echo "Old unrotated log files found in $dir:" | mail -s "Log Rotation Alert" "$ALERT_EMAIL"
echo "$old_files" | mail -s "Log Rotation Alert" "$ALERT_EMAIL"
fi
fi
done
```
Monitoring and Maintenance
Regular Maintenance Tasks
1. Review Rotation Effectiveness: Regularly check if rotation is preventing disk space issues.
2. Update Retention Policies: Adjust retention based on compliance requirements and disk usage patterns.
3. Monitor System Performance: Ensure rotation isn't impacting system performance.
Creating Rotation Reports
```bash
#!/bin/bash
Log rotation report generator
REPORT_FILE="/var/log/rotation-report.txt"
DATE=$(date '+%Y-%m-%d %H:%M:%S')
echo "Log Rotation Report - $DATE" > "$REPORT_FILE"
echo "=================================" >> "$REPORT_FILE"
Check logrotate status
echo -e "\nLogrotate Status:" >> "$REPORT_FILE"
cat /var/lib/logrotate/status | head -20 >> "$REPORT_FILE"
Disk usage summary
echo -e "\nDisk Usage Summary:" >> "$REPORT_FILE"
df -h /var/log >> "$REPORT_FILE"
Recent rotation activity
echo -e "\nRecent Rotation Activity:" >> "$REPORT_FILE"
find /var/log -name "*.gz" -mtime -1 -ls >> "$REPORT_FILE"
Journal usage (if systemd)
if command -v journalctl > /dev/null; then
echo -e "\nJournal Usage:" >> "$REPORT_FILE"
journalctl --disk-usage >> "$REPORT_FILE"
fi
echo "Report generated: $REPORT_FILE"
```
Health Check Script
```bash
#!/bin/bash
Log rotation health check
ISSUES_FOUND=0
Check if logrotate is installed and working
if ! command -v logrotate > /dev/null; then
echo "ERROR: logrotate is not installed"
ISSUES_FOUND=1
fi
Check configuration syntax
if ! logrotate -d /etc/logrotate.conf > /dev/null 2>&1; then
echo "ERROR: logrotate configuration has syntax errors"
ISSUES_FOUND=1
fi
Check disk space
DISK_USAGE=$(df /var/log | awk 'NR==2 {print $5}' | sed 's/%//')
if [ "$DISK_USAGE" -gt 80 ]; then
echo "WARNING: /var/log is ${DISK_USAGE}% full"
ISSUES_FOUND=1
fi
Check for stuck log files
LARGE_LOGS=$(find /var/log -name "*.log" -size +100M 2>/dev/null)
if [ -n "$LARGE_LOGS" ]; then
echo "WARNING: Large log files detected:"
echo "$LARGE_LOGS"
ISSUES_FOUND=1
fi
if [ $ISSUES_FOUND -eq 0 ]; then
echo "Log rotation health check: PASSED"
exit 0
else
echo "Log rotation health check: FAILED"
exit 1
fi
```
Conclusion
Log rotation is an essential aspect of Linux system administration that requires careful planning and implementation. This comprehensive guide has covered the fundamental concepts, practical implementations, and advanced techniques for managing log rotation in Linux environments.
Key Takeaways
1. Logrotate is the Standard: Use logrotate for most log rotation needs, as it's robust, well-tested, and widely supported.
2. Test Configurations: Always test rotation configurations before deploying to production systems.
3. Monitor and Maintain: Implement monitoring to ensure rotation is working effectively and adjust policies as needed.
4. Security Matters: Maintain proper permissions and consider integrity checking for sensitive logs.
5. Performance Impact: Consider the performance impact of rotation operations and schedule them appropriately.
Next Steps
After implementing log rotation:
1. Set up Monitoring: Implement alerts for rotation failures and disk space issues.
2. Document Procedures: Create documentation for your specific rotation policies and procedures.
3. Regular Reviews: Periodically review rotation effectiveness and adjust policies as needed.
4. Backup Strategy: Ensure rotated logs are included in your backup strategy.
5. Compliance Alignment: Verify that rotation policies meet your organization's compliance requirements.
Additional Resources
- Logrotate Manual: `man logrotate` for detailed option descriptions
- System Logs: Monitor `/var/log/syslog` and `journalctl` for rotation-related messages
- Community Forums: Linux distribution-specific forums for troubleshooting help
- Documentation: Vendor documentation for application-specific rotation requirements
By following the practices and examples outlined in this guide, you'll be able to implement robust log rotation strategies that maintain system performance while preserving important historical data for troubleshooting and compliance purposes.
Remember that log rotation is not a "set it and forget it" task—it requires ongoing monitoring and occasional adjustments to ensure optimal performance and reliability of your Linux systems.