How to show CPU usage → mpstat

How to Show CPU Usage → mpstat Table of Contents 1. [Introduction](#introduction) 2. [Prerequisites](#prerequisites) 3. [Understanding mpstat](#understanding-mpstat) 4. [Installation and Setup](#installation-and-setup) 5. [Basic mpstat Usage](#basic-mpstat-usage) 6. [Advanced mpstat Options](#advanced-mpstat-options) 7. [Interpreting mpstat Output](#interpreting-mpstat-output) 8. [Practical Examples and Use Cases](#practical-examples-and-use-cases) 9. [Troubleshooting Common Issues](#troubleshooting-common-issues) 10. [Best Practices and Professional Tips](#best-practices-and-professional-tips) 11. [Comparison with Other CPU Monitoring Tools](#comparison-with-other-cpu-monitoring-tools) 12. [Conclusion](#conclusion) Introduction CPU usage monitoring is a critical aspect of system administration and performance optimization. The `mpstat` command, part of the sysstat package, provides detailed multiprocessor statistics and real-time CPU usage information on Linux systems. This comprehensive guide will teach you how to effectively use mpstat to monitor CPU performance, understand system bottlenecks, and make informed decisions about system optimization. Whether you're a system administrator troubleshooting performance issues, a developer optimizing applications, or a DevOps engineer monitoring production systems, mastering mpstat will enhance your ability to analyze CPU utilization patterns and identify performance problems before they impact your users. Prerequisites Before diving into mpstat usage, ensure you have: - Operating System: Linux-based system (Ubuntu, CentOS, RHEL, Debian, etc.) - User Privileges: Standard user access (root privileges not required for basic usage) - Basic Knowledge: Familiarity with Linux command line interface - Terminal Access: SSH access or local terminal session - System Requirements: Modern Linux kernel (2.6 or later recommended) Required Packages The mpstat command is part of the sysstat package. Most Linux distributions include it by default, but you may need to install it manually. Understanding mpstat What is mpstat? The `mpstat` (Multi-Processor Statistics) command is a powerful system monitoring tool that displays activities for each available processor on a multiprocessor system. It provides real-time and historical CPU usage statistics, making it invaluable for: - Performance Monitoring: Track CPU utilization across all cores - Bottleneck Identification: Identify which processes consume the most CPU resources - Capacity Planning: Understand CPU usage patterns for scaling decisions - Troubleshooting: Diagnose performance issues and system slowdowns - Benchmarking: Compare system performance under different conditions Key Features - Real-time CPU statistics reporting - Per-processor and system-wide statistics - Historical data collection and analysis - Interrupt and soft interrupt monitoring - Integration with other sysstat tools - Scriptable output for automation Installation and Setup Installing sysstat Package On Ubuntu/Debian Systems ```bash sudo apt update sudo apt install sysstat ``` On CentOS/RHEL/Fedora Systems ```bash CentOS/RHEL 7 and earlier sudo yum install sysstat CentOS/RHEL 8+ and Fedora sudo dnf install sysstat ``` On SUSE/openSUSE Systems ```bash sudo zypper install sysstat ``` Verifying Installation After installation, verify mpstat is available: ```bash mpstat -V ``` This should display the sysstat version information, confirming successful installation. Enabling Data Collection To enable automatic data collection for historical analysis: ```bash sudo systemctl enable sysstat sudo systemctl start sysstat ``` Basic mpstat Usage Simple CPU Usage Display The most basic mpstat command shows current CPU usage: ```bash mpstat ``` Sample Output: ``` Linux 5.4.0-74-generic (hostname) 01/15/2024 _x86_64_ (4 CPU) 02:30:15 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 02:30:15 PM all 12.50 0.00 3.25 1.75 0.00 0.50 0.00 0.00 0.00 82.00 ``` Real-Time Monitoring To display CPU statistics every few seconds: ```bash Update every 2 seconds mpstat 2 Update every 5 seconds, show 10 reports mpstat 5 10 ``` Display All Processors Show statistics for each individual CPU core: ```bash mpstat -P ALL ``` Sample Output: ``` Linux 5.4.0-74-generic (hostname) 01/15/2024 _x86_64_ (4 CPU) 02:35:22 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 02:35:22 PM all 15.25 0.00 4.50 2.25 0.00 0.75 0.00 0.00 0.00 77.25 02:35:22 PM 0 18.50 0.00 5.25 1.75 0.00 1.00 0.00 0.00 0.00 73.50 02:35:22 PM 1 14.25 0.00 4.00 2.50 0.00 0.75 0.00 0.00 0.00 78.50 02:35:22 PM 2 12.75 0.00 3.75 2.75 0.00 0.50 0.00 0.00 0.00 80.25 02:35:22 PM 3 15.50 0.00 5.00 2.00 0.00 0.75 0.00 0.00 0.00 76.75 ``` Advanced mpstat Options Specific Processor Monitoring Monitor specific CPU cores: ```bash Monitor CPU 0 and 2 mpstat -P 0,2 2 Monitor CPU 1 only mpstat -P 1 3 5 ``` Interrupt Statistics Display interrupt-related statistics: ```bash Show interrupt statistics mpstat -I SUM 2 Show per-CPU interrupt statistics mpstat -I CPU 2 Show specific interrupt statistics mpstat -I SCPU 2 ``` JSON Output Format Generate JSON-formatted output for programmatic processing: ```bash mpstat -o JSON 2 3 ``` Historical Data Analysis Analyze historical CPU data from sar files: ```bash Analyze data from specific date mpstat -f /var/log/sysstat/saXX Show data from specific time range sar -f /var/log/sysstat/sa15 -s 10:00:00 -e 14:00:00 ``` Interpreting mpstat Output Understanding mpstat output is crucial for effective CPU monitoring. Here's a detailed breakdown of each column: CPU Usage Metrics | Metric | Description | Normal Range | Concern Level | |--------|-------------|--------------|---------------| | %usr | User space CPU usage | 0-70% | >80% sustained | | %nice | Nice priority processes | 0-10% | >20% | | %sys | System/kernel space usage | 0-30% | >40% sustained | | %iowait | I/O wait time | 0-20% | >30% | | %irq | Hardware interrupt handling | 0-5% | >10% | | %soft | Software interrupt handling | 0-10% | >15% | | %steal | Stolen time (virtualized) | 0-5% | >10% | | %guest | Guest OS CPU time | 0-50% | Context dependent | | %gnice | Nice guest processes | 0-10% | Context dependent | | %idle | Idle time | 20-90% | <10% sustained | Performance Indicators High CPU Utilization Signs - %usr > 80%: CPU-intensive applications or insufficient processing power - %sys > 40%: Kernel bottlenecks, excessive system calls, or driver issues - %iowait > 30%: Storage I/O bottlenecks affecting CPU performance - %idle < 10%: System operating at or near capacity Balanced System Indicators - %usr: 30-60%: Healthy application load - %sys: 5-20%: Normal system overhead - %iowait: 0-10%: Efficient I/O operations - %idle: 30-60%: Adequate headroom for peak loads Practical Examples and Use Cases Example 1: Basic System Monitoring Monitor overall system CPU usage with updates every 3 seconds: ```bash mpstat 3 ``` Use Case: General system health monitoring during normal operations. Example 2: Detailed Multi-Core Analysis Analyze individual CPU core performance: ```bash mpstat -P ALL 2 10 ``` Use Case: Identifying uneven load distribution across CPU cores or single-threaded application bottlenecks. Example 3: I/O Wait Investigation Focus on I/O wait times during database operations: ```bash mpstat 1 60 | grep -v "^$" | awk '{print $1, $2, $6}' ``` Use Case: Troubleshooting database performance issues or storage bottlenecks. Example 4: Production Monitoring Script Create a monitoring script for production environments: ```bash #!/bin/bash cpu_monitor.sh LOGFILE="/var/log/cpu_monitoring.log" THRESHOLD=80 while true; do TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S') CPU_USAGE=$(mpstat 1 1 | awk '/Average/ {print 100-$NF}') if (( $(echo "$CPU_USAGE > $THRESHOLD" | bc -l) )); then echo "$TIMESTAMP - HIGH CPU USAGE: ${CPU_USAGE}%" >> $LOGFILE # Send alert or take action fi sleep 60 done ``` Use Case: Automated monitoring with alerting for production systems. Example 5: Performance Benchmarking Compare CPU performance before and after optimization: ```bash Before optimization mpstat 1 300 > cpu_before.log After optimization mpstat 1 300 > cpu_after.log Analysis echo "Before optimization average CPU usage:" awk '/Average/ {print "User:", $3"% System:", $5"% Idle:", $NF"%"}' cpu_before.log echo "After optimization average CPU usage:" awk '/Average/ {print "User:", $3"% System:", $5"% Idle:", $NF"%"}' cpu_after.log ``` Use Case: Measuring the impact of system optimizations or application changes. Example 6: Virtualized Environment Monitoring Monitor CPU steal time in virtual machines: ```bash mpstat -P ALL 5 | awk '{if($9>5) print "High steal time on CPU", $2, ":", $9"%"}' ``` Use Case: Detecting hypervisor resource contention in virtualized environments. Troubleshooting Common Issues Issue 1: mpstat Command Not Found Problem: System reports "command not found" when running mpstat. Solution: ```bash Check if sysstat is installed dpkg -l | grep sysstat # Debian/Ubuntu rpm -qa | grep sysstat # CentOS/RHEL Install if missing sudo apt install sysstat # Debian/Ubuntu sudo yum install sysstat # CentOS/RHEL ``` Issue 2: No Historical Data Available Problem: Cannot access historical CPU statistics. Solution: ```bash Enable sysstat data collection sudo systemctl enable sysstat sudo systemctl start sysstat Verify configuration sudo cat /etc/default/sysstat # Debian/Ubuntu sudo cat /etc/sysconfig/sysstat # CentOS/RHEL Ensure ENABLED="true" ``` Issue 3: Inconsistent CPU Readings Problem: mpstat shows different values compared to other tools. Diagnosis: ```bash Compare with other tools mpstat 1 1 top -n 1 | head -3 cat /proc/loadavg ``` Solution: Different tools may use different calculation methods or sampling intervals. Use consistent measurement periods for accurate comparisons. Issue 4: High System CPU Usage Problem: Consistently high %sys values indicate kernel-level issues. Investigation: ```bash Check for high interrupt rates mpstat -I SUM 1 5 Identify problematic processes ps aux --sort=-%cpu | head -10 Check kernel messages dmesg | tail -20 ``` Issue 5: Permission Denied Errors Problem: Cannot access certain statistics or log files. Solution: ```bash Add user to appropriate groups sudo usermod -a -G adm $USER For system statistics access sudo chmod +r /var/log/sysstat/* ``` Best Practices and Professional Tips Monitoring Strategy 1. Establish Baselines ```bash Create baseline measurements during different periods mpstat 5 720 > baseline_business_hours.log # 1 hour during peak mpstat 5 720 > baseline_off_hours.log # 1 hour during off-peak ``` 2. Set Appropriate Intervals - Real-time troubleshooting: 1-2 second intervals - General monitoring: 5-10 second intervals - Long-term analysis: 1-5 minute intervals - Capacity planning: 15-30 minute intervals 3. Focus on Relevant Metrics ```bash For web servers, focus on user and I/O wait mpstat 5 | awk '{print $1, $2, $3, $6, $NF}' For database servers, emphasize system and I/O wait mpstat 5 | awk '{print $1, $2, $5, $6, $NF}' ``` Automation and Scripting 1. Automated Alerting ```bash #!/bin/bash Advanced CPU monitoring with multiple thresholds HIGH_THRESHOLD=80 CRITICAL_THRESHOLD=95 CPU_USAGE=$(mpstat 1 1 | awk '/Average/ {print 100-$NF}' | cut -d. -f1) if [ $CPU_USAGE -gt $CRITICAL_THRESHOLD ]; then echo "CRITICAL: CPU usage at ${CPU_USAGE}%" | mail -s "CPU Alert" admin@company.com elif [ $CPU_USAGE -gt $HIGH_THRESHOLD ]; then echo "WARNING: CPU usage at ${CPU_USAGE}%" | logger -t cpu_monitor fi ``` 2. Data Collection and Analysis ```bash Collect comprehensive CPU statistics #!/bin/bash DATE=$(date +%Y%m%d) LOGDIR="/var/log/performance" mkdir -p $LOGDIR Collect 24 hours of data mpstat -P ALL 300 288 > $LOGDIR/cpu_detailed_$DATE.log & Collect interrupt statistics mpstat -I SUM 300 288 > $LOGDIR/interrupts_$DATE.log & ``` Performance Optimization 1. Load Balancing Verification ```bash Check CPU core utilization balance mpstat -P ALL 2 30 | awk '/Average/ && $2 != "all" {print "CPU"$2": "$NF"% idle"}' ``` 2. Application Performance Correlation ```bash Monitor CPU during specific application tests echo "Starting application performance test..." mpstat 1 > app_test_cpu.log & MPSTAT_PID=$! Run your application test here ./run_performance_test.sh kill $MPSTAT_PID echo "CPU monitoring completed. Check app_test_cpu.log" ``` Integration with Other Tools 1. Combine with iotop for comprehensive analysis ```bash Terminal 1: Monitor CPU mpstat -P ALL 2 Terminal 2: Monitor I/O iotop -o Terminal 3: Monitor memory free -h -s 2 ``` 2. Export data for visualization ```bash Generate CSV format for Excel/visualization tools mpstat -o JSON 5 720 | jq -r '.sysstat.hosts[0].statistics[] | [.timestamp.time, .["cpu-load"][0].usr, .["cpu-load"][0].sys, .["cpu-load"][0].idle] | @csv' > cpu_data.csv ``` Comparison with Other CPU Monitoring Tools mpstat vs. top | Feature | mpstat | top | |---------|--------|-----| | Real-time updates | Yes | Yes | | Per-CPU statistics | Excellent | Limited | | Process details | No | Yes | | Historical data | Yes | No | | Scriptable output | Excellent | Limited | | Resource usage | Low | Low | mpstat vs. htop | Feature | mpstat | htop | |---------|--------|-----| | Visual interface | Text-based | Interactive | | CPU breakdown | Detailed | Visual bars | | Process management | No | Yes | | Automation friendly | Excellent | Poor | | Multi-core display | Excellent | Good | mpstat vs. sar | Feature | mpstat | sar | |---------|--------|-----| | CPU focus | Specialized | General | | Data collection | Manual/scripted | Automatic | | Historical analysis | Good | Excellent | | Report variety | CPU-focused | Comprehensive | | Learning curve | Easy | Moderate | When to Use Each Tool - Use mpstat when: You need detailed CPU statistics, per-core analysis, or scriptable monitoring - Use top/htop when: You need to identify specific processes consuming CPU - Use sar when: You need comprehensive system monitoring with automatic data collection - Use iostat when: CPU issues are related to I/O bottlenecks Advanced Use Cases Container and Kubernetes Monitoring Monitor CPU usage in containerized environments: ```bash Monitor CPU usage for container workloads docker exec -it container_name mpstat -P ALL 2 Kubernetes pod CPU monitoring kubectl exec -it pod_name -- mpstat 2 10 ``` Cloud Environment Optimization Optimize cloud instance CPU utilization: ```bash #!/bin/bash Cloud instance right-sizing analysis DURATION=3600 # 1 hour monitoring INTERVAL=60 # 1 minute intervals echo "Starting CPU utilization analysis for cloud right-sizing..." mpstat $INTERVAL $(($DURATION/$INTERVAL)) | tee cpu_rightsizing.log Analysis AVG_CPU=$(awk '/Average/ {print 100-$NF}' cpu_rightsizing.log) echo "Average CPU utilization: $AVG_CPU%" if (( $(echo "$AVG_CPU < 20" | bc -l) )); then echo "Recommendation: Consider downsizing instance" elif (( $(echo "$AVG_CPU > 80" | bc -l) )); then echo "Recommendation: Consider upsizing instance" else echo "Recommendation: Current instance size appears appropriate" fi ``` Database Performance Monitoring Specialized monitoring for database servers: ```bash #!/bin/bash Database server CPU monitoring DB_BACKUP_TIME="02:00" DB_PEAK_TIME="10:00" Monitor during backup operations if [[ $(date +%H:%M) == $DB_BACKUP_TIME ]]; then echo "Monitoring CPU during database backup..." mpstat 30 120 > cpu_during_backup.log # 1 hour monitoring fi Monitor during peak business hours if [[ $(date +%H:%M) == $DB_PEAK_TIME ]]; then echo "Monitoring CPU during peak hours..." mpstat 10 360 > cpu_during_peak.log # 1 hour monitoring fi ``` Conclusion The `mpstat` command is an indispensable tool for Linux system administrators and performance engineers. Its ability to provide detailed, real-time CPU statistics across multiple processors makes it essential for: - System Performance Monitoring: Understanding CPU utilization patterns and identifying bottlenecks - Capacity Planning: Making informed decisions about hardware upgrades and resource allocation - Troubleshooting: Diagnosing performance issues and system slowdowns - Optimization: Measuring the impact of system tuning and application changes - Automation: Integrating CPU monitoring into automated monitoring and alerting systems Key Takeaways 1. Master the Basics: Start with simple `mpstat` commands and gradually incorporate advanced options 2. Understand the Metrics: Learn what each CPU usage metric indicates and their normal ranges 3. Establish Baselines: Create performance baselines during different operational periods 4. Automate Monitoring: Implement scripted monitoring for proactive system management 5. Correlate Data: Combine mpstat with other monitoring tools for comprehensive system analysis 6. Practice Regularly: Regular use in different scenarios builds expertise and intuition Next Steps To further enhance your system monitoring capabilities: 1. Explore Related Tools: Learn `sar`, `iostat`, and `pidstat` from the sysstat package 2. Implement Monitoring Solutions: Set up comprehensive monitoring with tools like Nagios, Zabbix, or Prometheus 3. Study Performance Tuning: Understand how to optimize systems based on CPU usage patterns 4. Develop Automation: Create scripts and tools that leverage mpstat for automated system management 5. Learn Visualization: Use tools like Grafana to create dashboards from mpstat data By mastering mpstat and integrating it into your system administration toolkit, you'll be well-equipped to maintain optimal system performance and quickly resolve performance-related issues. Remember that effective system monitoring is an ongoing process that requires consistent attention and continuous learning. The investment in understanding CPU monitoring through mpstat will pay dividends in system reliability, performance optimization, and your overall effectiveness as a system administrator or DevOps professional.