How to monitor system performance in Linux
How to Monitor System Performance in Linux
Monitoring system performance is crucial for maintaining optimal Linux server and desktop operations. Whether you're troubleshooting slow applications, planning capacity upgrades, or ensuring system stability, understanding how to monitor your Linux system's performance metrics is an essential skill for administrators and users alike.
This comprehensive guide covers the essential tools and techniques for monitoring CPU usage, memory consumption, disk I/O, network activity, and overall system health in Linux environments.
Why System Performance Monitoring Matters
Performance monitoring helps you:
- Identify resource bottlenecks before they impact users
- Optimize system configurations for better efficiency
- Plan hardware upgrades based on actual usage data
- Troubleshoot performance issues quickly
- Ensure applications run within acceptable parameters
- Monitor security threats through unusual activity patterns
Essential Built-in Linux Performance Monitoring Tools
Using `top` Command for Real-time Process Monitoring
The `top` command is the most fundamental tool for monitoring system performance in real-time. It displays running processes, CPU usage, memory consumption, and system load.
```bash
top
```
Understanding top Output:
- PID: Process ID
- USER: Process owner
- PR: Process priority
- NI: Nice value
- VIRT: Virtual memory used
- RES: Resident memory used
- SHR: Shared memory
- %CPU: CPU usage percentage
- %MEM: Memory usage percentage
- TIME+: Total CPU time used
- COMMAND: Process name
Useful top Commands:
```bash
Sort by CPU usage
top -o %CPU
Sort by memory usage
top -o %MEM
Show processes for specific user
top -u username
Update interval every 2 seconds
top -d 2
```
Enhanced Process Monitoring with `htop`
`htop` provides a more user-friendly and colorful interface compared to `top`. If not installed, install it using:
```bash
Ubuntu/Debian
sudo apt install htop
CentOS/RHEL/Fedora
sudo yum install htop
or
sudo dnf install htop
```
Launch htop:
```bash
htop
```
htop Advantages:
- Color-coded display
- Mouse support
- Easy process filtering
- Tree view of processes
- Built-in process management
Monitoring CPU Performance
Using `vmstat` for System Statistics
`vmstat` provides information about processes, memory, paging, block I/O, and CPU activity.
```bash
Display current statistics
vmstat
Update every 2 seconds, 5 times
vmstat 2 5
```
Key vmstat Columns:
- r: Number of processes waiting for CPU
- b: Number of processes in uninterruptible sleep
- us: User CPU time percentage
- sy: System CPU time percentage
- id: Idle CPU time percentage
- wa: I/O wait time percentage
Monitoring CPU Usage with `iostat`
Install the sysstat package to use iostat:
```bash
Ubuntu/Debian
sudo apt install sysstat
CentOS/RHEL/Fedora
sudo yum install sysstat
```
Monitor CPU statistics:
```bash
Display CPU usage
iostat -c
Update every 2 seconds
iostat -c 2
Display extended statistics
iostat -x 2
```
Using `sar` for Historical Performance Data
The `sar` (System Activity Reporter) command collects and displays system performance data.
```bash
Display current CPU usage
sar -u
Show CPU usage every 2 seconds, 5 times
sar -u 2 5
Display historical data from specific date
sar -u -f /var/log/sysstat/saXX
```
Memory Performance Monitoring
Checking Memory Usage with `free`
The `free` command displays memory usage information:
```bash
Display memory usage in human-readable format
free -h
Show memory usage in MB
free -m
Display memory usage every 2 seconds
free -h -s 2
```
Understanding free Output:
- total: Total installed memory
- used: Used memory
- free: Unused memory
- shared: Memory used by tmpfs
- buff/cache: Buffer and cache memory
- available: Memory available for applications
Advanced Memory Analysis with `/proc/meminfo`
For detailed memory information:
```bash
cat /proc/meminfo
```
Key metrics to monitor:
```bash
Check specific memory metrics
grep -E "(MemTotal|MemFree|MemAvailable|Buffers|Cached)" /proc/meminfo
```
Disk I/O Performance Monitoring
Using `iotop` for Disk I/O Monitoring
Install iotop to monitor disk I/O by processes:
```bash
Ubuntu/Debian
sudo apt install iotop
CentOS/RHEL/Fedora
sudo yum install iotop
```
Monitor disk I/O:
```bash
Display I/O usage by processes
sudo iotop
Show only processes with I/O activity
sudo iotop -o
Display accumulated I/O
sudo iotop -a
```
Monitoring Disk Usage with `df` and `du`
Check disk space usage:
```bash
Display filesystem usage
df -h
Show inode usage
df -i
Check specific directory size
du -sh /path/to/directory
Find largest directories
du -h /home | sort -hr | head -10
```
Advanced Disk Monitoring with `iostat`
Monitor disk I/O statistics:
```bash
Display device utilization
iostat -d 2
Show extended statistics
iostat -dx 2
Monitor specific device
iostat -dx sda 2
```
Key iostat Metrics:
- %util: Device utilization percentage
- r/s: Read requests per second
- w/s: Write requests per second
- await: Average wait time for I/O requests
Network Performance Monitoring
Using `iftop` for Network Bandwidth Monitoring
Install iftop:
```bash
Ubuntu/Debian
sudo apt install iftop
CentOS/RHEL/Fedora
sudo yum install iftop
```
Monitor network connections:
```bash
Monitor default interface
sudo iftop
Monitor specific interface
sudo iftop -i eth0
Show port numbers
sudo iftop -P
```
Network Statistics with `netstat` and `ss`
Check network connections and statistics:
```bash
Display all connections
netstat -tuln
Show network statistics
netstat -s
Modern alternative to netstat
ss -tuln
Display socket statistics
ss -s
```
Monitoring Network Traffic with `vnstat`
Install vnstat for network traffic statistics:
```bash
Ubuntu/Debian
sudo apt install vnstat
Configure for interface
sudo vnstat -u -i eth0
Display statistics
vnstat
vnstat -d # Daily stats
vnstat -m # Monthly stats
```
System Load and Performance Overview
Understanding System Load Average
Monitor system load with `uptime`:
```bash
uptime
```
The load average shows:
- 1-minute load average
- 5-minute load average
- 15-minute load average
Load Average Guidelines:
- Load equal to CPU cores = 100% utilization
- Load higher than CPU cores = system overloaded
- Load lower than CPU cores = system underutilized
Using `w` Command for User Activity
Check logged-in users and their activity:
```bash
w
```
Advanced Performance Monitoring Tools
Installing and Using `glances`
Glances provides a comprehensive system monitoring interface:
```bash
Ubuntu/Debian
sudo apt install glances
CentOS/RHEL/Fedora
sudo yum install glances
Run glances
glances
```
Glances Features:
- CPU, memory, disk, and network monitoring
- Process monitoring
- System information
- Web interface option
- Alert system
Using `nmon` for Performance Analysis
Install nmon:
```bash
Ubuntu/Debian
sudo apt install nmon
Run nmon
nmon
```
Press keys to enable different views:
- `c`: CPU usage
- `m`: Memory usage
- `d`: Disk I/O
- `n`: Network statistics
Creating Performance Monitoring Scripts
Basic System Health Check Script
Create a simple monitoring script:
```bash
#!/bin/bash
system_health.sh
echo "=== System Health Report ==="
echo "Date: $(date)"
echo ""
echo "=== System Load ==="
uptime
echo ""
echo "=== Memory Usage ==="
free -h
echo ""
echo "=== Disk Usage ==="
df -h | grep -v tmpfs
echo ""
echo "=== Top 5 CPU Processes ==="
ps aux --sort=-%cpu | head -6
echo ""
echo "=== Top 5 Memory Processes ==="
ps aux --sort=-%mem | head -6
```
Make it executable and run:
```bash
chmod +x system_health.sh
./system_health.sh
```
Automated Monitoring with Cron
Schedule regular monitoring:
```bash
Edit crontab
crontab -e
Add entry to run every 5 minutes
/5 * /path/to/system_health.sh >> /var/log/system_health.log 2>&1
```
Troubleshooting Common Performance Issues
Identifying High CPU Usage
When experiencing high CPU usage:
1. Identify the culprit process:
```bash
top -o %CPU
```
2. Check for system processes:
```bash
ps aux | grep -E "(kworker|ksoftirqd)"
```
3. Monitor CPU per core:
```bash
htop
# Press 'F2' → Display options → Detailed CPU time
```
Resolving Memory Issues
For memory-related problems:
1. Check memory usage:
```bash
free -h
cat /proc/meminfo | grep -i available
```
2. Identify memory-hungry processes:
```bash
ps aux --sort=-%mem | head -10
```
3. Check for memory leaks:
```bash
# Monitor specific process
watch -n 1 'ps -p PID -o pid,ppid,cmd,%mem,%cpu'
```
Diagnosing Disk I/O Bottlenecks
When experiencing slow disk performance:
1. Monitor I/O wait:
```bash
iostat -x 1
# Look for high %iowait values
```
2. Identify I/O intensive processes:
```bash
sudo iotop -ao
```
3. Check disk health:
```bash
sudo smartctl -a /dev/sda
```
Performance Monitoring Best Practices
Establishing Baselines
1. Document normal performance metrics during low usage periods
2. Record performance during peak usage times
3. Monitor trends over weeks and months
4. Use tools like sar to collect historical data
Setting Up Alerts
Create monitoring scripts with thresholds:
```bash
#!/bin/bash
alert_script.sh
CPU_THRESHOLD=80
MEMORY_THRESHOLD=90
DISK_THRESHOLD=85
CPU_USAGE=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | sed 's/%us,//')
MEMORY_USAGE=$(free | grep Mem | awk '{printf("%.0f", $3/$2 * 100.0)}')
DISK_USAGE=$(df / | tail -1 | awk '{print $5}' | sed 's/%//')
if [ ${CPU_USAGE%.*} -gt $CPU_THRESHOLD ]; then
echo "High CPU usage: ${CPU_USAGE}%" | mail -s "CPU Alert" admin@example.com
fi
if [ $MEMORY_USAGE -gt $MEMORY_THRESHOLD ]; then
echo "High memory usage: ${MEMORY_USAGE}%" | mail -s "Memory Alert" admin@example.com
fi
if [ $DISK_USAGE -gt $DISK_THRESHOLD ]; then
echo "High disk usage: ${DISK_USAGE}%" | mail -s "Disk Alert" admin@example.com
fi
```
Regular Maintenance Tasks
1. Clean up log files regularly
2. Monitor disk space usage
3. Update system packages
4. Review and optimize startup services
5. Perform regular system reboots for updates
Conclusion
Effective Linux system performance monitoring requires understanding and utilizing various built-in tools and commands. From basic utilities like `top` and `free` to advanced tools like `glances` and `nmon`, each serves specific monitoring needs.
Key takeaways for successful performance monitoring:
- Use multiple tools for comprehensive monitoring
- Establish performance baselines for your systems
- Automate monitoring tasks with scripts and cron jobs
- Set up alerting for critical thresholds
- Monitor trends over time rather than just current values
- Combine different metrics for accurate diagnosis
Regular performance monitoring helps maintain system reliability, optimize resource usage, and prevent issues before they impact users. Start with basic tools like `top`, `free`, and `iostat`, then gradually incorporate more advanced monitoring solutions as your needs grow.
By implementing these monitoring practices, you'll be well-equipped to maintain optimal Linux system performance and quickly identify and resolve any performance-related issues that arise.