How to understand system logs in /var/log
How to Understand System Logs in /var/log
Table of Contents
1. [Introduction](#introduction)
2. [Prerequisites](#prerequisites)
3. [Understanding the /var/log Directory Structure](#understanding-the-varlog-directory-structure)
4. [Essential Log Files and Their Purposes](#essential-log-files-and-their-purposes)
5. [Reading and Interpreting Log Entries](#reading-and-interpreting-log-entries)
6. [Tools for Log Analysis](#tools-for-log-analysis)
7. [Common Log Analysis Scenarios](#common-log-analysis-scenarios)
8. [Advanced Log Management Techniques](#advanced-log-management-techniques)
9. [Troubleshooting Common Issues](#troubleshooting-common-issues)
10. [Best Practices for Log Management](#best-practices-for-log-management)
11. [Conclusion](#conclusion)
Introduction
System logs are the backbone of Linux system administration and troubleshooting. Located primarily in the `/var/log` directory, these files contain crucial information about system events, errors, security incidents, and application behavior. Understanding how to read and interpret these logs is essential for maintaining system health, diagnosing problems, and ensuring security.
This comprehensive guide will teach you everything you need to know about system logs in `/var/log`, from basic log file structures to advanced analysis techniques. Whether you're a system administrator, developer, or Linux enthusiast, mastering log analysis will significantly enhance your troubleshooting capabilities and system understanding.
Prerequisites
Before diving into log analysis, ensure you have:
- Basic Linux command-line knowledge: Familiarity with commands like `ls`, `cat`, `grep`, `tail`, and `head`
- Root or sudo access: Many log files require elevated privileges to read
- Text editor familiarity: Knowledge of editors like `vi`, `nano`, or `less`
- Basic understanding of Linux system architecture: Processes, services, and system components
- Time and date format comprehension: Understanding timestamps and time zones
Required Tools
Most tools for log analysis come pre-installed on Linux systems:
- `tail`, `head`, `cat` - Basic file viewing
- `grep`, `awk`, `sed` - Text processing
- `less`, `more` - Paging through files
- `journalctl` - systemd journal viewer (modern systems)
- `logrotate` - Log rotation management
Understanding the /var/log Directory Structure
The `/var/log` directory is the standard location for system and application log files in Unix-like operating systems. Let's explore its typical structure:
```bash
View the /var/log directory contents
ls -la /var/log/
Common directory structure
/var/log/
├── alternatives.log # Package alternatives changes
├── auth.log # Authentication attempts
├── boot.log # Boot process messages
├── cron.log # Cron job execution
├── daemon.log # System daemon messages
├── dpkg.log # Package management (Debian/Ubuntu)
├── faillog # Failed login attempts
├── kern.log # Kernel messages
├── lastlog # Last login information
├── mail.log # Mail server logs
├── messages # General system messages
├── secure # Security-related messages (Red Hat/CentOS)
├── syslog # System-wide messages
├── user.log # User-level messages
├── wtmp # Login/logout records
├── apache2/ # Apache web server logs
├── mysql/ # MySQL database logs
└── journal/ # systemd journal files
```
Log File Permissions and Ownership
Log files typically have restricted permissions for security reasons:
```bash
Check log file permissions
ls -la /var/log/auth.log
-rw-r----- 1 syslog adm 15234 Nov 15 10:30 /var/log/auth.log
Common permission patterns:
640 (rw-r-----): Owner can read/write, group can read
644 (rw-r--r--): Owner can read/write, others can read
600 (rw-------): Only owner can read/write
```
Essential Log Files and Their Purposes
System-Wide Logs
1. /var/log/syslog
The primary system log containing messages from various system components:
```bash
View recent syslog entries
tail -f /var/log/syslog
Example entries:
Nov 15 10:30:15 server01 systemd[1]: Started Daily apt download activities.
Nov 15 10:30:20 server01 CRON[12345]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
```
2. /var/log/messages
General system messages (more common on Red Hat-based systems):
```bash
View system messages
tail -20 /var/log/messages
Example content:
Nov 15 10:25:30 server01 kernel: [12345.678901] USB disconnect, address 1
Nov 15 10:25:35 server01 NetworkManager[1234]: device (eth0): carrier is ON
```
3. /var/log/kern.log
Kernel-specific messages including hardware issues and driver problems:
```bash
Monitor kernel messages
tail -f /var/log/kern.log
Example kernel messages:
Nov 15 10:20:15 server01 kernel: [12340.123456] ata1: SATA link up 6.0 Gbps
Nov 15 10:20:16 server01 kernel: [12341.234567] EXT4-fs (sda1): mounted filesystem
```
Security and Authentication Logs
1. /var/log/auth.log
Authentication attempts, sudo usage, and security events:
```bash
View authentication logs
sudo tail -20 /var/log/auth.log
Example entries:
Nov 15 10:15:30 server01 sshd[12345]: Accepted publickey for user01 from 192.168.1.100
Nov 15 10:16:45 server01 sudo: user01 : TTY=pts/0 ; PWD=/home/user01 ; USER=root ; COMMAND=/bin/cat /var/log/auth.log
Nov 15 10:17:00 server01 sshd[12350]: Failed password for invalid user hacker from 203.0.113.1
```
2. /var/log/secure (Red Hat/CentOS)
Similar to auth.log but used on Red Hat-based systems:
```bash
View security logs on Red Hat systems
sudo tail /var/log/secure
```
3. /var/log/faillog
Failed login attempts in binary format:
```bash
View failed login attempts
sudo faillog -a
```
Service-Specific Logs
1. /var/log/apache2/ or /var/log/httpd/
Web server logs including access and error logs:
```bash
Apache access log
tail -f /var/log/apache2/access.log
Example:
192.168.1.100 - - [15/Nov/2023:10:30:15 +0000] "GET /index.html HTTP/1.1" 200 1234
Apache error log
tail -f /var/log/apache2/error.log
Example:
[Wed Nov 15 10:30:20.123456 2023] [error] [pid 12345] [client 192.168.1.100] File does not exist: /var/www/html/missing.html
```
2. /var/log/mysql/
MySQL database server logs:
```bash
MySQL error log
sudo tail /var/log/mysql/error.log
MySQL slow query log (if enabled)
sudo tail /var/log/mysql/mysql-slow.log
```
3. /var/log/cron.log
Cron job execution logs:
```bash
View cron execution logs
tail -20 /var/log/cron.log
Example entries:
Nov 15 10:30:01 server01 CRON[12345]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Nov 15 10:35:01 server01 CRON[12350]: (user01) CMD (/home/user01/scripts/backup.sh)
```
Reading and Interpreting Log Entries
Standard Log Entry Format
Most log entries follow a common format:
```
TIMESTAMP HOSTNAME SERVICE[PID]: MESSAGE
```
Let's break down a typical log entry:
```bash
Nov 15 10:30:15 server01 sshd[12345]: Accepted publickey for user01 from 192.168.1.100 port 54321 ssh2
```
- Nov 15 10:30:15: Timestamp (Month Day Hour:Minute:Second)
- server01: Hostname of the system
- sshd: Service or daemon name
- [12345]: Process ID (PID)
- Accepted publickey...: The actual log message
Log Levels and Priorities
Linux systems use standard log levels defined by syslog:
| Level | Name | Description |
|-------|------|-------------|
| 0 | Emergency | System is unusable |
| 1 | Alert | Action must be taken immediately |
| 2 | Critical | Critical conditions |
| 3 | Error | Error conditions |
| 4 | Warning | Warning conditions |
| 5 | Notice | Normal but significant condition |
| 6 | Info | Informational messages |
| 7 | Debug | Debug-level messages |
Timestamp Interpretation
Understanding timestamps is crucial for log analysis:
```bash
Standard format: Mon DD HH:MM:SS
Nov 15 10:30:15
Some logs include year and timezone:
2023-11-15T10:30:15.123456+00:00
Convert timestamps if needed
date -d "Nov 15 10:30:15" "+%Y-%m-%d %H:%M:%S"
```
Tools for Log Analysis
Basic Command-Line Tools
1. tail - View Recent Log Entries
```bash
View last 20 lines
tail -20 /var/log/syslog
Follow log in real-time
tail -f /var/log/syslog
Follow multiple logs simultaneously
tail -f /var/log/syslog /var/log/auth.log
Show last N lines from multiple files
tail -n 50 /var/log/syslog /var/log/kern.log
```
2. head - View Beginning of Log Files
```bash
View first 10 lines
head /var/log/syslog
View first 30 lines
head -30 /var/log/syslog
```
3. grep - Search and Filter Logs
```bash
Search for specific terms
grep "error" /var/log/syslog
Case-insensitive search
grep -i "failed" /var/log/auth.log
Search for multiple patterns
grep -E "error|warning|critical" /var/log/syslog
Show line numbers
grep -n "SSH" /var/log/auth.log
Show context around matches
grep -C 3 "kernel panic" /var/log/kern.log
Invert match (show lines NOT containing pattern)
grep -v "INFO" /var/log/application.log
```
4. awk - Advanced Text Processing
```bash
Extract specific columns (timestamp and message)
awk '{print $1, $2, $3, $6}' /var/log/syslog
Filter by time range
awk '/Nov 15 10:3[0-9]/' /var/log/syslog
Count occurrences
awk '/failed/ {count++} END {print "Failed attempts:", count}' /var/log/auth.log
Extract IP addresses from auth.log
awk '/Failed password/ {print $(NF-3)}' /var/log/auth.log | sort | uniq -c
```
5. sed - Stream Editor for Filtering
```bash
Remove timestamps for cleaner output
sed 's/^[A-Za-z] [0-9] [0-9]:[0-9]:[0-9] [a-zA-Z0-9] //' /var/log/syslog
Extract lines between specific times
sed -n '/10:30:00/,/10:35:00/p' /var/log/syslog
```
Advanced Log Analysis Tools
1. journalctl - systemd Journal Viewer
Modern Linux systems use systemd with its own logging system:
```bash
View all journal entries
journalctl
Follow journal in real-time
journalctl -f
Show logs for specific service
journalctl -u ssh.service
Show logs for specific time range
journalctl --since "2023-11-15 10:00:00" --until "2023-11-15 11:00:00"
Show logs for specific priority level
journalctl -p err
Show logs for specific user
journalctl _UID=1000
Show kernel messages only
journalctl -k
Show logs in JSON format
journalctl -o json
Show disk usage of journal
journalctl --disk-usage
```
2. less - Advanced File Paging
```bash
View log with search capabilities
less /var/log/syslog
Search within less:
/pattern - search forward
?pattern - search backward
n - next match
N - previous match
G - go to end of file
g - go to beginning of file
```
3. zcat and zgrep - Compressed Log Analysis
Many systems compress old logs to save space:
```bash
View compressed logs
zcat /var/log/syslog.1.gz
Search in compressed logs
zgrep "error" /var/log/syslog.*.gz
Combine multiple compressed logs
zcat /var/log/syslog.*.gz | grep "specific_pattern"
```
Common Log Analysis Scenarios
Scenario 1: Investigating Failed Login Attempts
```bash
Find all failed SSH login attempts
grep "Failed password" /var/log/auth.log
Count failed attempts by IP address
grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr
Find successful logins after failed attempts
grep -A 5 -B 5 "Failed password" /var/log/auth.log | grep "Accepted"
Check for brute force patterns
awk '/Failed password/ {ip=$(NF-3); count[ip]++} END {for (i in count) if (count[i] > 10) print i, count[i]}' /var/log/auth.log
```
Scenario 2: Debugging System Boot Issues
```bash
View boot messages
journalctl -b
View previous boot
journalctl -b -1
Check kernel messages during boot
dmesg | grep -i error
Look for service startup failures
journalctl -b | grep -i failed
Check specific service status
journalctl -u networking.service -b
```
Scenario 3: Monitoring System Performance
```bash
Check for out-of-memory events
grep -i "out of memory" /var/log/kern.log
Look for disk space issues
grep -i "no space left" /var/log/syslog
Monitor CPU-intensive processes
grep -i "cpu" /var/log/syslog | tail -20
Check for hardware errors
grep -i "hardware error" /var/log/kern.log
```
Scenario 4: Web Server Troubleshooting
```bash
Find 404 errors in Apache logs
grep " 404 " /var/log/apache2/access.log
Count requests by IP address
awk '{print $1}' /var/log/apache2/access.log | sort | uniq -c | sort -nr | head -10
Find slow requests (if using combined log format)
awk '$NF > 5000000 {print $0}' /var/log/apache2/access.log
Check for PHP errors
grep -i "php" /var/log/apache2/error.log
```
Scenario 5: Database Performance Analysis
```bash
Check MySQL slow queries
sudo tail -50 /var/log/mysql/mysql-slow.log
Find MySQL connection errors
grep -i "connection" /var/log/mysql/error.log
Monitor MySQL restarts
grep -i "mysqld.*started" /var/log/syslog
```
Advanced Log Management Techniques
Log Rotation and Archival
Understanding logrotate configuration:
```bash
View logrotate configuration
cat /etc/logrotate.conf
Check service-specific configurations
ls /etc/logrotate.d/
Example logrotate configuration for custom application:
cat > /etc/logrotate.d/myapp << EOF
/var/log/myapp/*.log {
daily
missingok
rotate 52
compress
delaycompress
notifempty
create 644 myapp myapp
postrotate
systemctl reload myapp
endscript
}
EOF
Test logrotate configuration
logrotate -d /etc/logrotate.d/myapp
Force log rotation
logrotate -f /etc/logrotate.d/myapp
```
Real-time Log Monitoring
Create monitoring scripts for critical events:
```bash
#!/bin/bash
Script: monitor_auth.sh
Monitor authentication logs for suspicious activity
tail -f /var/log/auth.log | while read line; do
if echo "$line" | grep -q "Failed password"; then
ip=$(echo "$line" | awk '{print $(NF-3)}')
echo "$(date): Failed login attempt from $ip"
# Count recent failures from this IP
recent_failures=$(grep "Failed password.*$ip" /var/log/auth.log | tail -10 | wc -l)
if [ "$recent_failures" -gt 5 ]; then
echo "WARNING: Multiple failed attempts from $ip"
# Could add automatic blocking here
fi
fi
done
```
Log Aggregation and Analysis
For multiple servers, consider centralized logging:
```bash
Configure rsyslog to send logs to central server
echo ". @@logserver.example.com:514" >> /etc/rsyslog.conf
Configure rsyslog to receive logs (on log server)
echo '$ModLoad imudp' >> /etc/rsyslog.conf
echo '$UDPServerRun 514' >> /etc/rsyslog.conf
echo '$UDPServerAddress 0.0.0.0' >> /etc/rsyslog.conf
Restart rsyslog
systemctl restart rsyslog
```
Custom Log Parsing Scripts
Create specialized parsing tools:
```bash
#!/bin/bash
Script: parse_apache_logs.sh
Parse Apache logs for specific information
LOG_FILE="/var/log/apache2/access.log"
echo "=== Apache Log Analysis ==="
echo "Top 10 IP addresses:"
awk '{print $1}' "$LOG_FILE" | sort | uniq -c | sort -nr | head -10
echo -e "\nTop 10 requested pages:"
awk '{print $7}' "$LOG_FILE" | sort | uniq -c | sort -nr | head -10
echo -e "\nHTTP status code distribution:"
awk '{print $9}' "$LOG_FILE" | sort | uniq -c | sort -nr
echo -e "\nTraffic by hour:"
awk '{print $4}' "$LOG_FILE" | cut -d: -f2 | sort | uniq -c
```
Troubleshooting Common Issues
Issue 1: Log Files Not Updating
Symptoms: Log files appear to be static or not receiving new entries.
Troubleshooting Steps:
```bash
Check if logging service is running
systemctl status rsyslog
systemctl status systemd-journald
Verify log file permissions
ls -la /var/log/syslog
Check disk space
df -h /var/log
Test logging manually
logger "Test message from $(whoami)"
tail /var/log/syslog | grep "Test message"
Check rsyslog configuration
rsyslogd -N1 -f /etc/rsyslog.conf
```
Issue 2: Cannot Access Log Files
Symptoms: Permission denied errors when trying to read logs.
Solutions:
```bash
Check current user permissions
groups
Add user to adm group (for log access)
sudo usermod -a -G adm username
Use sudo for privileged logs
sudo tail /var/log/auth.log
Check file permissions and ownership
ls -la /var/log/auth.log
Fix permissions if necessary (be careful!)
sudo chown syslog:adm /var/log/auth.log
sudo chmod 640 /var/log/auth.log
```
Issue 3: Log Files Growing Too Large
Symptoms: Disk space issues due to large log files.
Solutions:
```bash
Check log file sizes
du -sh /var/log/*
Identify largest log files
find /var/log -type f -exec du -h {} + | sort -rh | head -20
Configure logrotate for problematic logs
sudo nano /etc/logrotate.d/custom-app
Force immediate rotation
sudo logrotate -f /etc/logrotate.d/custom-app
Manually compress old logs
gzip /var/log/large-application.log.old
```
Issue 4: Missing Log Entries
Symptoms: Expected log entries are not appearing.
Troubleshooting:
```bash
Check application logging configuration
grep -r "log" /etc/myapp/
Verify syslog facility and priority settings
grep "myapp" /etc/rsyslog.conf
Test application logging
strace -e write -p $(pgrep myapp) 2>&1 | grep -i log
Check if logs are going to journal instead
journalctl -u myapp.service
Verify log level configuration
cat /etc/myapp/logging.conf
```
Issue 5: Timestamp Issues
Symptoms: Log timestamps are incorrect or inconsistent.
Solutions:
```bash
Check system time and timezone
date
timedatectl status
Set correct timezone
sudo timedatectl set-timezone America/New_York
Synchronize with NTP
sudo systemctl enable ntp
sudo systemctl start ntp
Check NTP synchronization
ntpq -p
Verify hardware clock
sudo hwclock --show
```
Best Practices for Log Management
Security Considerations
1. Protect Log Files:
```bash
Set appropriate permissions
sudo chmod 640 /var/log/sensitive.log
sudo chown root:adm /var/log/sensitive.log
Consider encrypting sensitive logs
gpg --symmetric /var/log/sensitive.log
```
2. Log Integrity:
```bash
Create checksums for important logs
sha256sum /var/log/auth.log > /var/log/auth.log.sha256
Verify integrity
sha256sum -c /var/log/auth.log.sha256
```
3. Centralized Logging:
- Use tools like ELK Stack (Elasticsearch, Logstash, Kibana)
- Implement syslog forwarding to secure central servers
- Consider using TLS for log transmission
Performance Optimization
1. Efficient Log Rotation:
```bash
Configure appropriate rotation schedules
Daily for high-volume logs
Weekly for moderate-volume logs
Monthly for low-volume logs
Example optimized logrotate config:
/var/log/high-volume.log {
hourly
rotate 24
compress
delaycompress
missingok
notifempty
copytruncate
}
```
2. Log Level Management:
```bash
Adjust log levels based on environment
Production: WARNING and above
Development: DEBUG and above
Testing: INFO and above
```
3. Storage Optimization:
```bash
Use compression for archived logs
find /var/log -name ".log." -not -name "*.gz" -exec gzip {} \;
Implement log retention policies
find /var/log -name "*.log.gz" -mtime +90 -delete
```
Monitoring and Alerting
1. Automated Monitoring:
```bash
#!/bin/bash
Script: log_monitor.sh
Monitor logs for critical events
CRITICAL_PATTERNS=(
"kernel panic"
"out of memory"
"file system.*read-only"
"failed.*times"
)
for pattern in "${CRITICAL_PATTERNS[@]}"; do
if grep -q "$pattern" /var/log/syslog; then
echo "CRITICAL: Found '$pattern' in system logs"
# Send alert (email, Slack, etc.)
fi
done
```
2. Log Analysis Automation:
```bash
Create daily log summary reports
#!/bin/bash
Script: daily_log_report.sh
echo "Daily Log Summary - $(date)"
echo "================================"
echo "Authentication Summary:"
grep "$(date '+%b %d')" /var/log/auth.log | \
awk '/Accepted/ {success++} /Failed/ {failed++} END {
print "Successful logins:", success+0
print "Failed logins:", failed+0
}'
echo -e "\nTop Error Messages:"
grep "$(date '+%b %d')" /var/log/syslog | \
grep -i error | \
awk '{for(i=6;i<=NF;i++) printf $i" "; print ""}' | \
sort | uniq -c | sort -nr | head -5
echo -e "\nSystem Load Events:"
grep "$(date '+%b %d')" /var/log/syslog | \
grep -E "(high load|cpu|memory)" | wc -l
```
Documentation and Procedures
1. Log Location Documentation:
Create a reference document listing all log files and their purposes:
```
Application Logs:
- /var/log/myapp/application.log - Main application events
- /var/log/myapp/error.log - Application errors
- /var/log/myapp/access.log - User access logs
System Logs:
- /var/log/syslog - General system messages
- /var/log/auth.log - Authentication events
- /var/log/kern.log - Kernel messages
```
2. Incident Response Procedures:
Document standard log analysis procedures for common incidents:
- Security breach investigation
- Performance issue diagnosis
- Service outage analysis
Conclusion
Understanding system logs in `/var/log` is a fundamental skill for Linux system administration and troubleshooting. This comprehensive guide has covered:
- Directory structure and file types in `/var/log`
- Essential log files and their specific purposes
- Reading and interpreting log entry formats and timestamps
- Command-line tools for effective log analysis
- Real-world scenarios and practical examples
- Advanced techniques for log management and automation
- Troubleshooting approaches for common log-related issues
- Best practices for security, performance, and maintenance
Key Takeaways
1. Start with the basics: Master fundamental tools like `tail`, `grep`, and `less` before moving to advanced techniques
2. Understand log formats: Familiarize yourself with timestamp formats and log entry structures
3. Use appropriate tools: Choose between traditional syslog files and systemd journal based on your system
4. Implement proper rotation: Configure logrotate to prevent disk space issues
5. Monitor proactively: Set up automated monitoring for critical events
6. Maintain security: Protect log files and consider centralized logging for important systems
Next Steps
To further develop your log analysis skills:
1. Practice regularly: Set up a test environment and generate various log scenarios
2. Learn advanced tools: Explore ELK Stack, Splunk, or other log management platforms
3. Automate analysis: Create custom scripts for your specific monitoring needs
4. Study application logs: Learn about logs specific to your applications and services
5. Implement centralized logging: Set up log aggregation for multiple systems
6. Develop incident response procedures: Create standardized approaches for log-based troubleshooting
Remember that effective log analysis is both an art and a science. The more you practice with real-world scenarios, the better you'll become at quickly identifying issues and understanding system behavior through log examination. System logs are your window into what's happening on your Linux systems – master this skill, and you'll be well-equipped to maintain robust, secure, and well-performing systems.