How to Report System Activity → sar Table of Contents - [Introduction](#introduction) - [Prerequisites](#prerequisites) - [Understanding sar](#understanding-sar) - [Installation and Setup](#installation-and-setup) - [Basic sar Usage](#basic-sar-usage) - [Monitoring Different System Resources](#monitoring-different-system-resources) - [Working with Historical Data](#working-with-historical-data) - [Advanced sar Techniques](#advanced-sar-techniques) - [Practical Examples and Use Cases](#practical-examples-and-use-cases) - [Troubleshooting Common Issues](#troubleshooting-common-issues) - [Best Practices](#best-practices) - [Integration with Other Tools](#integration-with-other-tools) - [Conclusion](#conclusion) Introduction System Activity Reporter (sar) is one of the most powerful and comprehensive tools available for monitoring Linux system performance. Part of the sysstat package, sar provides detailed insights into various aspects of system activity, including CPU utilization, memory usage, disk I/O, network activity, and much more. Whether you're a system administrator troubleshooting performance issues or a developer optimizing application performance, understanding how to effectively use sar is essential for maintaining healthy Linux systems. This comprehensive guide will walk you through everything you need to know about using sar, from basic monitoring commands to advanced analysis techniques. You'll learn how to interpret sar output, collect historical data, and implement monitoring strategies that will help you proactively identify and resolve system performance issues. Prerequisites Before diving into sar usage, ensure you have the following: System Requirements - Linux-based operating system (Ubuntu, CentOS, RHEL, Debian, etc.) - Root or sudo access for installation and certain monitoring tasks - Basic understanding of Linux command line interface - Familiarity with system performance concepts (CPU, memory, I/O) Knowledge Prerequisites - Understanding of Linux file system structure - Basic knowledge of system processes and resources - Familiarity with cron jobs (for automated data collection) - Understanding of system performance metrics Understanding sar What is sar? The System Activity Reporter (sar) is a command-line utility that collects, reports, and saves system activity information. It's part of the sysstat package and provides a comprehensive view of system performance metrics over time. Unlike real-time monitoring tools, sar excels at historical analysis and trend identification. Key Features Comprehensive Monitoring: sar can monitor CPU usage, memory utilization, disk I/O, network activity, swap usage, and more. Historical Data: Automatically collects and stores performance data for historical analysis. Flexible Reporting: Offers numerous options for customizing reports and output formats. Automated Collection: Integrates with cron for automated data collection at regular intervals. Multiple Output Formats: Supports various output formats including plain text, CSV, and binary formats. How sar Works sar operates in two primary modes: 1. Real-time Monitoring: Displays current system activity as it happens 2. Historical Reporting: Analyzes previously collected data stored in log files The tool uses the `/proc` filesystem to gather system statistics and can either display them immediately or store them in binary format for later analysis. Installation and Setup Installing sysstat Package On Ubuntu/Debian Systems ```bash sudo apt update sudo apt install sysstat ``` On CentOS/RHEL/Fedora Systems ```bash CentOS/RHEL 7/8 sudo yum install sysstat CentOS/RHEL 9/Fedora sudo dnf install sysstat ``` On SUSE Systems ```bash sudo zypper install sysstat ``` Enabling Data Collection After installation, you need to enable automatic data collection: Ubuntu/Debian ```bash sudo systemctl enable sysstat sudo systemctl start sysstat ``` Edit the configuration file: ```bash sudo nano /etc/default/sysstat ``` Change `ENABLED="false"` to `ENABLED="true"` CentOS/RHEL/Fedora ```bash sudo systemctl enable sysstat sudo systemctl start sysstat ``` Configuring Data Collection Intervals The default configuration collects data every 10 minutes. To modify this: ```bash sudo nano /etc/cron.d/sysstat ``` Example cron entry for 5-minute intervals: ```bash /5 * root /usr/lib64/sa/sa1 1 1 ``` Verifying Installation Check if sar is working correctly: ```bash sar -u 1 3 ``` This command should display CPU utilization for 3 intervals of 1 second each. Basic sar Usage Command Syntax The basic syntax for sar is: ```bash sar [options] [interval] [count] ``` - options: Specify what to monitor (CPU, memory, disk, etc.) - interval: Time between samples in seconds - count: Number of samples to collect Essential sar Options | Option | Description | Example | |--------|-------------|---------| | `-u` | CPU utilization | `sar -u 2 5` | | `-r` | Memory utilization | `sar -r 1 10` | | `-d` | Disk activity | `sar -d 5 3` | | `-n` | Network statistics | `sar -n DEV 1 5` | | `-b` | I/O and transfer rate | `sar -b 2 4` | | `-S` | Swap space utilization | `sar -S 1 3` | | `-q` | Queue length and load averages | `sar -q 5 2` | Real-time Monitoring Examples Monitor CPU Usage ```bash sar -u 2 5 ``` This displays CPU utilization every 2 seconds for 5 samples: ``` Linux 5.4.0-42-generic (hostname) 01/15/2024 _x86_64_ (4 CPU) 02:30:15 PM CPU %user %nice %system %iowait %steal %idle 02:30:17 PM all 5.25 0.00 2.01 0.25 0.00 92.49 02:30:19 PM all 3.76 0.00 1.51 0.00 0.00 94.73 02:30:21 PM all 4.52 0.00 1.76 0.50 0.00 93.22 02:30:23 PM all 6.28 0.00 2.26 0.00 0.00 91.46 02:30:25 PM all 4.77 0.00 1.88 0.25 0.00 93.10 Average: all 4.92 0.00 1.88 0.20 0.00 93.00 ``` Monitor Memory Usage ```bash sar -r 1 3 ``` Output shows memory statistics: ``` 02:35:10 PM kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 02:35:11 PM 1048576 3145728 2097152 66.67 524288 1572864 1048576 33.33 1048576 524288 64 02:35:12 PM 1048320 3145472 2097408 66.68 524288 1572864 1048576 33.33 1048640 524224 68 02:35:13 PM 1048064 3145216 2097664 66.69 524352 1572800 1048640 33.34 1048704 524160 72 Average: 1048320 3145472 2097408 66.68 524309 1572843 1048597 33.33 1048640 524224 68 ``` Monitoring Different System Resources CPU Monitoring Basic CPU Statistics ```bash sar -u 1 5 ``` Per-CPU Statistics ```bash sar -P ALL 1 3 ``` This shows statistics for each individual CPU core: ``` 02:40:15 PM CPU %user %nice %system %iowait %steal %idle 02:40:16 PM all 4.25 0.00 1.75 0.00 0.00 94.00 02:40:16 PM 0 3.96 0.00 1.98 0.00 0.00 94.06 02:40:16 PM 1 4.95 0.00 1.98 0.00 0.00 93.07 02:40:16 PM 2 3.92 0.00 1.96 0.00 0.00 94.12 02:40:16 PM 3 4.17 0.00 0.00 0.00 0.00 95.83 ``` Understanding CPU Metrics - %user: Time spent in user mode - %nice: Time spent in user mode with nice priority - %system: Time spent in kernel mode - %iowait: Time spent waiting for I/O operations - %steal: Time stolen by hypervisor (virtualized environments) - %idle: Time spent idle Memory Monitoring Memory Utilization ```bash sar -r 2 3 ``` Swap Usage ```bash sar -S 2 3 ``` Memory Paging Statistics ```bash sar -B 1 5 ``` Disk I/O Monitoring Block Device Statistics ```bash sar -d 2 4 ``` Specific Device Monitoring ```bash sar -d -p 1 5 ``` The `-p` flag shows device names instead of device numbers: ``` 02:45:10 PM DEV tps rkB/s wkB/s areq-sz aqu-sz await svctm %util 02:45:11 PM sda 15.84 63.37 126.73 12.00 0.02 1.25 0.79 1.25 02:45:11 PM sdb 8.91 35.64 71.29 12.00 0.01 0.89 0.56 0.50 ``` Understanding Disk Metrics - tps: Transfers per second - rkB/s: Kilobytes read per second - wkB/s: Kilobytes written per second - areq-sz: Average request size - aqu-sz: Average queue size - await: Average wait time - svctm: Average service time - %util: Percentage of CPU time during which I/O requests were issued Network Monitoring Network Device Statistics ```bash sar -n DEV 1 5 ``` Network Error Statistics ```bash sar -n EDEV 1 3 ``` TCP Statistics ```bash sar -n TCP 2 4 ``` Socket Statistics ```bash sar -n SOCK 1 3 ``` Working with Historical Data Understanding sar Data Files sar stores historical data in binary files located in `/var/log/sa/`. Files are named using the format `saDD` where DD is the day of the month. Viewing Available Data Files ```bash ls -la /var/log/sa/ ``` Output: ``` -rw-r--r-- 1 root root 876544 Jan 15 23:59 sa15 -rw-r--r-- 1 root root 854016 Jan 14 23:59 sa14 -rw-r--r-- 1 root root 831488 Jan 13 23:59 sa13 ``` Analyzing Historical Data View Yesterday's CPU Data ```bash sar -u -f /var/log/sa/sa14 ``` View Specific Time Range ```bash sar -u -s 09:00:00 -e 17:00:00 -f /var/log/sa/sa15 ``` This shows CPU utilization between 9 AM and 5 PM for the 15th day. Generate Daily Reports ```bash sar -A -f /var/log/sa/sa15 ``` The `-A` option displays all available statistics for the specified day. Creating Custom Reports Export Data to CSV Format ```bash sadf -d /var/log/sa/sa15 -- -u > cpu_usage.csv ``` Generate XML Output ```bash sadf -x /var/log/sa/sa15 -- -r > memory_usage.xml ``` Create JSON Format Reports ```bash sadf -j /var/log/sa/sa15 -- -d > disk_activity.json ``` Advanced sar Techniques Monitoring Specific Processes Process Creation and Context Switches ```bash sar -w 1 5 ``` Process Queue Statistics ```bash sar -q 2 3 ``` Advanced Network Monitoring Monitor Specific Network Interface ```bash sar -n DEV,EDEV 1 5 | grep eth0 ``` Network File System Statistics ```bash sar -n NFS 2 4 ``` IP Statistics ```bash sar -n IP 1 3 ``` System Load Analysis Comprehensive System Overview ```bash sar -A 5 2 ``` Custom Monitoring Script Create a script for comprehensive monitoring: ```bash #!/bin/bash comprehensive_monitor.sh DATE=$(date +%Y%m%d_%H%M%S) LOGDIR="/var/log/custom_monitoring" mkdir -p $LOGDIR CPU monitoring sar -u 1 60 > $LOGDIR/cpu_$DATE.log & Memory monitoring sar -r 1 60 > $LOGDIR/memory_$DATE.log & Disk monitoring sar -d 1 60 > $LOGDIR/disk_$DATE.log & Network monitoring sar -n DEV 1 60 > $LOGDIR/network_$DATE.log & wait echo "Monitoring completed. Logs saved to $LOGDIR" ``` Make it executable and run: ```bash chmod +x comprehensive_monitor.sh ./comprehensive_monitor.sh ``` Practical Examples and Use Cases Performance Troubleshooting Scenarios Scenario 1: High CPU Usage Investigation ```bash Check current CPU usage sar -u 1 10 Check per-CPU statistics sar -P ALL 1 5 Check process queue and load averages sar -q 1 5 Analyze historical data for patterns sar -u -s 00:00:00 -e 23:59:59 -f /var/log/sa/sa$(date +%d) ``` Scenario 2: Memory Pressure Analysis ```bash Monitor memory utilization sar -r 2 10 Check swap usage sar -S 2 10 Monitor paging activity sar -B 1 15 Check for memory pressure patterns sar -r -s 08:00:00 -e 18:00:00 -f /var/log/sa/sa$(date +%d) ``` Scenario 3: Disk I/O Bottleneck Detection ```bash Monitor all block devices sar -d -p 1 10 Focus on specific high-usage device sar -d -p 1 20 | grep sda Check I/O wait times sar -u 1 10 | grep -E "(Average|iowait)" Historical disk analysis sar -d -p -s 09:00:00 -e 17:00:00 -f /var/log/sa/sa$(date +%d) ``` Capacity Planning Weekly Performance Summary ```bash #!/bin/bash weekly_summary.sh echo "Weekly Performance Summary" echo "==========================" for day in {01..07}; do if [ -f "/var/log/sa/sa$day" ]; then echo "Day $day:" echo "Average CPU Usage:" sar -u -f /var/log/sa/sa$day | tail -1 echo "Peak Memory Usage:" sar -r -f /var/log/sa/sa$day | awk 'NR>3 {if ($4>max) max=$4} END {print "Peak Memory Used: " max "%"}' echo "---" fi done ``` Automated Alerting Create Performance Alert Script ```bash #!/bin/bash performance_alert.sh CPU_THRESHOLD=80 MEMORY_THRESHOLD=90 DISK_UTIL_THRESHOLD=85 Check CPU usage CPU_USAGE=$(sar -u 1 1 | tail -1 | awk '{print 100-$8}') if (( $(echo "$CPU_USAGE > $CPU_THRESHOLD" | bc -l) )); then echo "ALERT: High CPU usage: ${CPU_USAGE}%" | mail -s "CPU Alert" admin@example.com fi Check memory usage MEMORY_USAGE=$(sar -r 1 1 | tail -1 | awk '{print $4}') if (( $(echo "$MEMORY_USAGE > $MEMORY_THRESHOLD" | bc -l) )); then echo "ALERT: High memory usage: ${MEMORY_USAGE}%" | mail -s "Memory Alert" admin@example.com fi Check disk utilization DISK_UTIL=$(sar -d 1 1 | tail -1 | awk '{print $10}') if (( $(echo "$DISK_UTIL > $DISK_UTIL_THRESHOLD" | bc -l) )); then echo "ALERT: High disk utilization: ${DISK_UTIL}%" | mail -s "Disk Alert" admin@example.com fi ``` Troubleshooting Common Issues sar Command Not Found Problem: Command not found error when running sar. Solution: ```bash Check if sysstat is installed which sar Install sysstat if missing sudo apt install sysstat # Ubuntu/Debian sudo yum install sysstat # CentOS/RHEL ``` No Data Collection Problem: sar shows no historical data or "Cannot open" errors. Solution: ```bash Check if data collection is enabled sudo systemctl status sysstat Enable data collection sudo systemctl enable sysstat sudo systemctl start sysstat Check cron jobs sudo crontab -l | grep sa1 Manually trigger data collection sudo /usr/lib64/sa/sa1 1 1 ``` Permission Denied Errors Problem: Cannot access sar data files. Solution: ```bash Check file permissions ls -la /var/log/sa/ Fix permissions if necessary sudo chmod 644 /var/log/sa/sa* Check if user is in appropriate group groups $USER ``` Incomplete or Corrupted Data Problem: sar data appears incomplete or corrupted. Solution: ```bash Verify data file integrity sadf -c /var/log/sa/sa$(date +%d) Remove corrupted files and restart collection sudo rm /var/log/sa/sa$(date +%d) sudo systemctl restart sysstat Check disk space for log directory df -h /var/log/ ``` High System Load from sar Problem: sar itself is causing high system load. Solution: ```bash Reduce collection frequency sudo nano /etc/cron.d/sysstat Change from /1 to /5 or */10 Limit data retention sudo nano /etc/sysstat/sysstat Set HISTORY=7 (keep 7 days instead of 28) Use nice priority for data collection sudo nice -n 10 /usr/lib64/sa/sa1 1 1 ``` Best Practices Data Collection Strategy Optimal Collection Intervals - Production systems: 5-10 minute intervals - Development systems: 10-15 minute intervals - Critical systems: 1-2 minute intervals during troubleshooting Data Retention Policy ```bash Configure in /etc/sysstat/sysstat HISTORY=28 # Keep 28 days of data COMPRESSAFTER=10 # Compress data older than 10 days ``` Monitoring Guidelines Essential Daily Checks ```bash #!/bin/bash daily_health_check.sh echo "Daily System Health Check - $(date)" echo "==================================" CPU utilization summary echo "CPU Utilization (last 24 hours):" sar -u -s $(date -d "1 day ago" +%H:%M:%S) | tail -1 Memory usage peak echo "Memory Usage:" sar -r -s $(date -d "1 day ago" +%H:%M:%S) | awk 'NR>3 {if ($4>max) max=$4} END {print "Peak: " max "%"}' Disk I/O summary echo "Disk I/O:" sar -d -s $(date -d "1 day ago" +%H:%M:%S) | tail -5 Network activity echo "Network Activity:" sar -n DEV -s $(date -d "1 day ago" +%H:%M:%S) | grep -E "(rxkB/s|txkB/s)" | tail -1 ``` Performance Baselines Establishing Baselines ```bash #!/bin/bash create_baseline.sh BASELINE_DIR="/var/log/baselines" mkdir -p $BASELINE_DIR Collect 1 week of data for baseline for i in {1..7}; do DAY=$(date -d "$i days ago" +%d) if [ -f "/var/log/sa/sa$DAY" ]; then # CPU baseline sar -u -f /var/log/sa/sa$DAY | tail -1 >> $BASELINE_DIR/cpu_baseline.txt # Memory baseline sar -r -f /var/log/sa/sa$DAY | awk 'END {print $4}' >> $BASELINE_DIR/memory_baseline.txt # Disk baseline sar -d -f /var/log/sa/sa$DAY | awk 'END {print $10}' >> $BASELINE_DIR/disk_baseline.txt fi done echo "Baseline data collected in $BASELINE_DIR" ``` Integration Strategies Log Management ```bash Rotate sar logs with logrotate sudo nano /etc/logrotate.d/sysstat /var/log/sa/sa?? { daily rotate 30 compress delaycompress missingok notifempty create 644 root root } ``` Monitoring Integration ```bash Export sar data to monitoring systems sadf -d /var/log/sa/sa$(date +%d) -- -u | \ awk -F';' 'NR>1 {print "cpu.user " $4 " " $3}' | \ nc monitoring-server 2003 ``` Integration with Other Tools Combining sar with Other Utilities sar + iostat for Comprehensive I/O Analysis ```bash #!/bin/bash io_analysis.sh echo "Comprehensive I/O Analysis" echo "=========================" echo "Block Device Statistics (sar):" sar -d -p 1 5 echo -e "\nExtended I/O Statistics (iostat):" iostat -x 1 5 echo -e "\nI/O Wait Analysis:" sar -u 1 5 | grep -E "(Average|%iowait)" ``` sar + vmstat for Memory Analysis ```bash #!/bin/bash memory_analysis.sh echo "Memory Analysis Report" echo "====================" echo "Memory Utilization (sar):" sar -r 1 3 echo -e "\nVirtual Memory Statistics (vmstat):" vmstat 1 3 echo -e "\nSwap Usage (sar):" sar -S 1 3 ``` Creating Dashboards Generate HTML Reports ```bash #!/bin/bash generate_html_report.sh HTML_FILE="/var/www/html/system_report.html" cat > $HTML_FILE << EOF

System Performance Report - $(date)

EOF echo "HTML report generated: $HTML_FILE" ``` Conclusion The sar command is an indispensable tool for Linux system administrators and performance engineers. Its comprehensive monitoring capabilities, combined with historical data analysis features, make it essential for maintaining optimal system performance and proactively identifying potential issues. Throughout this guide, we've covered everything from basic installation and configuration to advanced monitoring techniques and integration strategies. Key takeaways include: Comprehensive Monitoring: sar provides detailed insights into CPU, memory, disk, network, and other system resources, enabling holistic performance analysis. Historical Analysis: The ability to analyze historical data makes sar invaluable for trend identification, capacity planning, and root cause analysis. Flexibility: With numerous options and output formats, sar can be adapted to meet specific monitoring requirements and integrated with existing tools and workflows. Automation: sar's integration with cron and its scriptable nature enable automated monitoring and alerting solutions. Next Steps To further enhance your system monitoring capabilities: 1. Implement Regular Monitoring: Set up automated sar data collection and establish regular review processes 2. Create Custom Scripts: Develop monitoring scripts tailored to your specific environment and requirements 3. Establish Baselines: Create performance baselines to identify deviations and trends 4. Integration: Integrate sar with your existing monitoring infrastructure and alerting systems 5. Documentation: Document your monitoring procedures and create runbooks for common performance issues Additional Resources For continued learning and advanced techniques: - Explore the sysstat package documentation for additional utilities like iostat, mpstat, and pidstat - Learn about complementary monitoring tools like htop, iotop, and netstat - Study system performance tuning techniques to act on sar findings - Investigate enterprise monitoring solutions that can consume sar data By mastering sar and implementing the practices outlined in this guide, you'll be well-equipped to maintain high-performing Linux systems and quickly identify and resolve performance issues as they arise. Remember that effective system monitoring is an ongoing process that requires regular attention and continuous refinement of your monitoring strategies.