How to show CPU/mem stats → vmstat, mpstat, pidstat, dstat

How to Show CPU/Memory Stats: Complete Guide to vmstat, mpstat, pidstat, and dstat Table of Contents 1. [Introduction](#introduction) 2. [Prerequisites](#prerequisites) 3. [Understanding System Statistics](#understanding-system-statistics) 4. [vmstat - Virtual Memory Statistics](#vmstat---virtual-memory-statistics) 5. [mpstat - Multi-Processor Statistics](#mpstat---multi-processor-statistics) 6. [pidstat - Process Statistics](#pidstat---process-statistics) 7. [dstat - Versatile System Statistics](#dstat---versatile-system-statistics) 8. [Practical Examples and Use Cases](#practical-examples-and-use-cases) 9. [Troubleshooting Common Issues](#troubleshooting-common-issues) 10. [Best Practices and Tips](#best-practices-and-tips) 11. [Conclusion](#conclusion) Introduction System monitoring is a critical aspect of Linux administration that enables you to understand system performance, identify bottlenecks, and troubleshoot issues effectively. This comprehensive guide explores four essential command-line tools for monitoring CPU and memory statistics: `vmstat`, `mpstat`, `pidstat`, and `dstat`. Each tool offers unique perspectives on system performance, from high-level system overviews to detailed per-process analysis. By mastering these utilities, you'll gain the ability to diagnose performance problems, optimize system resources, and maintain healthy server environments. Throughout this article, you'll learn practical command usage, interpret output data, understand performance metrics, and discover advanced techniques for system monitoring and analysis. Prerequisites Before diving into these monitoring tools, ensure you have: - Linux System Access: Root or sudo privileges for comprehensive monitoring - Basic Command Line Knowledge: Familiarity with terminal operations - System Utilities: Most tools are pre-installed, but some may require installation - Understanding of System Concepts: Basic knowledge of processes, memory, and CPU architecture Installation Requirements Most modern Linux distributions include these tools by default. If missing, install them using: Ubuntu/Debian: ```bash sudo apt-get update sudo apt-get install sysstat dstat ``` RHEL/CentOS/Fedora: ```bash sudo yum install sysstat dstat or for newer versions sudo dnf install sysstat dstat ``` Understanding System Statistics Before exploring specific tools, it's essential to understand key performance metrics: CPU Metrics - User Time (us): Time spent executing user processes - System Time (sy): Time spent in kernel mode - Idle Time (id): Time when CPU is not processing - Wait Time (wa): Time waiting for I/O operations - Steal Time (st): Time stolen by hypervisor in virtualized environments Memory Metrics - Free Memory: Available RAM for new processes - Used Memory: Currently allocated RAM - Buffer/Cache: Memory used for disk caching - Swap Usage: Virtual memory usage on disk vmstat - Virtual Memory Statistics `vmstat` provides a comprehensive overview of system performance, including processes, memory, paging, block I/O, traps, and CPU activity. Basic vmstat Usage ```bash Display current statistics vmstat Update every 2 seconds, 5 times vmstat 2 5 Show statistics in megabytes vmstat -S M Display active/inactive memory vmstat -a ``` Understanding vmstat Output ```bash $ vmstat 2 3 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 0 1024768 85532 2847236 0 0 45 67 156 298 8 3 88 1 0 1 0 0 1023456 85532 2847240 0 0 0 18 189 356 5 2 93 0 0 0 0 0 1023456 85532 2847240 0 0 0 0 167 287 3 1 96 0 0 ``` Column Explanations Processes (procs): - `r`: Running processes - `b`: Blocked processes waiting for I/O Memory: - `swpd`: Virtual memory used (KB) - `free`: Free memory (KB) - `buff`: Buffer memory (KB) - `cache`: Cache memory (KB) Swap: - `si`: Memory swapped from disk (KB/s) - `so`: Memory swapped to disk (KB/s) I/O: - `bi`: Blocks received from device (blocks/s) - `bo`: Blocks sent to device (blocks/s) System: - `in`: Interrupts per second - `cs`: Context switches per second CPU: - `us`: User CPU time percentage - `sy`: System CPU time percentage - `id`: Idle CPU time percentage - `wa`: Wait time percentage - `st`: Stolen time percentage Advanced vmstat Options ```bash Display disk statistics vmstat -d Show partition statistics vmstat -p /dev/sda1 Display slab info vmstat -m Show fork statistics vmstat -f ``` mpstat - Multi-Processor Statistics `mpstat` provides detailed CPU statistics for multiprocessor systems, allowing per-CPU analysis and system-wide monitoring. Basic mpstat Usage ```bash Show current CPU statistics mpstat Update every 2 seconds, 5 times mpstat 2 5 Display all processors individually mpstat -P ALL 2 3 Show specific CPU (CPU 0) mpstat -P 0 2 3 ``` Understanding mpstat Output ```bash $ mpstat -P ALL 2 3 Linux 5.4.0-74-generic (server01) 06/15/2024 _x86_64_ (4 CPU) 02:30:15 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 02:30:17 PM all 12.56 0.00 3.77 0.25 0.00 0.50 0.00 0.00 0.00 82.92 02:30:17 PM 0 15.00 0.00 4.50 0.50 0.00 1.00 0.00 0.00 0.00 79.00 02:30:17 PM 1 10.50 0.00 3.50 0.00 0.00 0.00 0.00 0.00 0.00 86.00 02:30:17 PM 2 14.00 0.00 3.50 0.50 0.00 0.50 0.00 0.00 0.00 81.50 02:30:17 PM 3 10.50 0.00 3.50 0.00 0.00 0.50 0.00 0.00 0.00 85.50 ``` mpstat Metrics Explained - %usr: User-level applications CPU usage - %nice: Nice priority processes CPU usage - %sys: System-level (kernel) CPU usage - %iowait: CPU idle time waiting for I/O - %irq: Hardware interrupt handling time - %soft: Software interrupt handling time - %steal: Time stolen by hypervisor - %guest: Virtual CPU time for guest OS - %gnice: Nice priority guest processes - %idle: CPU idle time Advanced mpstat Features ```bash Show interrupts per CPU mpstat -I CPU 2 3 Display specific interrupt statistics mpstat -I SCPU 2 3 Show USB device statistics mpstat -u 2 3 JSON output format mpstat -o JSON 2 3 ``` pidstat - Process Statistics `pidstat` provides detailed statistics for individual processes, including CPU usage, memory consumption, and I/O activity. Basic pidstat Usage ```bash Show CPU statistics for all processes pidstat Update every 2 seconds, 3 times pidstat 2 3 Show specific process by PID pidstat -p 1234 Show processes by command name pidstat -C "apache" ``` CPU Statistics with pidstat ```bash $ pidstat 2 2 Linux 5.4.0-74-generic (server01) 06/15/2024 _x86_64_ (4 CPU) 02:35:10 PM UID PID %usr %system %guest %wait %CPU CPU Command 02:35:12 PM 0 1 0.00 0.50 0.00 0.00 0.50 1 systemd 02:35:12 PM 0 10 0.00 0.50 0.00 0.00 0.50 0 rcu_sched 02:35:12 PM 1000 1234 8.50 1.50 0.00 0.00 10.00 2 apache2 02:35:12 PM 1000 1235 7.00 1.00 0.00 0.50 8.00 3 apache2 ``` Memory Statistics with pidstat ```bash Show memory usage statistics pidstat -r 2 3 Example output 02:40:15 PM UID PID minflt/s majflt/s VSZ RSS %MEM Command 02:40:17 PM 0 1 0.00 0.00 225484 9156 0.04 systemd 02:40:17 PM 1000 1234 12.50 0.00 385648 45632 0.18 apache2 02:40:17 PM 1000 1235 10.00 0.00 385648 42108 0.17 apache2 ``` I/O Statistics with pidstat ```bash Show I/O statistics pidstat -d 2 3 Example output 02:45:20 PM UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command 02:45:22 PM 0 123 0.00 45.50 0.00 0 kworker 02:45:22 PM 1000 1234 12.00 8.50 0.00 2 apache2 02:45:22 PM 1001 5678 125.50 85.00 0.00 15 mysql ``` Advanced pidstat Options ```bash Show all statistics together pidstat -urd 2 3 Show thread statistics pidstat -t 2 3 Show child process statistics pidstat -T CHILD 2 3 Filter by user pidstat -U postgres 2 3 Show page faults and memory pidstat -r -p ALL 2 3 ``` dstat - Versatile System Statistics `dstat` is a versatile tool that combines features from vmstat, iostat, netstat, and ifstat, providing colorful and flexible system monitoring. Basic dstat Usage ```bash Basic system overview dstat Custom interval (2 seconds) dstat 2 Show specific statistics dstat -c -m -d -n Colorful output with time dstat -T ``` Understanding dstat Output ```bash $ dstat -cmdngy ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- usr sys idl wai hiq siq| read writ| recv send| in out | int csw 8 3 88 1 0 0| 45k 67k| 0 0 | 0 0 | 156 298 5 2 93 0 0 0| 0 18k| 156B 298B| 0 0 | 189 356 3 1 96 0 0 0| 0 0 | 287B 167B| 0 0 | 167 287 ``` Comprehensive dstat Monitoring ```bash CPU, memory, disk, network, and system stats dstat -cmdn Include swap and paging information dstat -cmdnsp Show top CPU and memory processes dstat -c -m --top-cpu --top-mem Network statistics with interface details dstat -n -N eth0,lo Disk statistics for specific devices dstat -d -D sda,sdb ``` Advanced dstat Features ```bash Show interrupts and context switches dstat --ipc Display filesystem statistics dstat --fs Show TCP connection statistics dstat --tcp Monitor specific processes dstat --proc-count Show battery information (laptops) dstat --battery Custom output to CSV file dstat -cmdngy --output system_stats.csv 5 ``` dstat Color Coding dstat uses color coding to highlight performance issues: - Green: Normal/good performance - Yellow: Warning levels - Red: Critical/high usage - Cyan: Information display Practical Examples and Use Cases Scenario 1: High CPU Usage Investigation ```bash Step 1: Check overall system load vmstat 1 10 Step 2: Identify which CPUs are busy mpstat -P ALL 1 5 Step 3: Find processes consuming CPU pidstat -u 1 5 | head -20 Step 4: Monitor specific high-usage process pidstat -p 1234 1 10 ``` Scenario 2: Memory Leak Detection ```bash Monitor memory usage over time vmstat -S M 5 60 Track memory usage by process pidstat -r 5 12 Watch for growing RSS values pidstat -r -p 1234 10 30 Use dstat for comprehensive view dstat -m --top-mem 5 20 ``` Scenario 3: I/O Bottleneck Analysis ```bash Check I/O wait times vmstat 2 10 Monitor disk I/O by process pidstat -d 2 10 Use dstat for disk and CPU correlation dstat -cdm 2 15 Focus on specific processes pidstat -d -p 1234,5678 2 10 ``` Scenario 4: System Performance Baseline ```bash Create comprehensive baseline dstat -cmdngy --output baseline_$(date +%Y%m%d).csv 60 1440 CPU baseline mpstat 60 24 > cpu_baseline_$(date +%Y%m%d).log Process baseline pidstat -u -r -d 300 288 > process_baseline_$(date +%Y%m%d).log ``` Scenario 5: Real-time Monitoring Dashboard ```bash Terminal 1: Overall system view watch -n 1 'vmstat 1 1' Terminal 2: CPU per core watch -n 2 'mpstat -P ALL 1 1' Terminal 3: Top processes watch -n 3 'pidstat -u 1 1 | head -15' Terminal 4: Comprehensive dstat dstat -cmdngy 2 ``` Troubleshooting Common Issues Issue 1: Command Not Found Problem: `mpstat: command not found` or similar errors. Solution: ```bash Install sysstat package sudo apt-get install sysstat # Ubuntu/Debian sudo yum install sysstat # RHEL/CentOS Verify installation which mpstat pidstat ``` Issue 2: Permission Denied Problem: Cannot access certain process information. Solution: ```bash Run with sudo for full access sudo pidstat -u 1 5 Or add user to appropriate groups sudo usermod -a -G adm username ``` Issue 3: High System Load but Low CPU Usage Problem: Load average is high, but CPU usage appears normal. Investigation: ```bash Check I/O wait vmstat 1 10 Look for blocked processes vmstat 1 5 | awk '{print $2}' # b column Identify I/O intensive processes pidstat -d 1 10 ``` Issue 4: Memory Statistics Don't Add Up Problem: Free + used memory doesn't equal total memory. Explanation: ```bash Check buffer/cache usage vmstat -S M Use -a flag for active/inactive memory vmstat -a -S M Modern systems use available memory concept cat /proc/meminfo | grep -E 'MemTotal|MemFree|MemAvailable' ``` Issue 5: Inconsistent Statistics Between Tools Problem: Different tools show varying statistics. Resolution: - Tools sample at different intervals - Some tools show instantaneous values, others show averages - Buffer/cache treatment may differ - Always compare tools with similar sampling intervals Issue 6: dstat Installation Issues Problem: dstat not available on newer distributions. Alternative Solutions: ```bash Use newer alternatives sudo apt-get install sysstat iotop htop Or compile from source wget http://dag.wieers.com/rpm/packages/dstat/dstat-0.7.4.tar.gz tar -xzf dstat-0.7.4.tar.gz cd dstat-0.7.4 sudo make install ``` Best Practices and Tips Monitoring Strategy 1. Establish Baselines: Record normal system behavior during different periods 2. Use Multiple Tools: Combine tools for comprehensive analysis 3. Monitor Trends: Focus on trends rather than instantaneous values 4. Document Findings: Keep logs of performance investigations Sampling Intervals ```bash Short intervals for immediate issues vmstat 1 60 Medium intervals for trend analysis mpstat 5 720 # 1 hour Long intervals for baseline creation dstat 300 288 # 24 hours ``` Automation and Scripting ```bash #!/bin/bash Performance monitoring script LOGDIR="/var/log/performance" DATE=$(date +%Y%m%d_%H%M%S) mkdir -p $LOGDIR Collect system stats vmstat 5 12 > $LOGDIR/vmstat_$DATE.log & mpstat -P ALL 5 12 > $LOGDIR/mpstat_$DATE.log & pidstat -u -r -d 5 12 > $LOGDIR/pidstat_$DATE.log & wait echo "Performance data collected in $LOGDIR" ``` Performance Thresholds CPU Usage Guidelines: - Normal: <70% average usage - Warning: 70-85% sustained usage - Critical: >85% sustained usage Memory Usage Guidelines: - Normal: <80% of physical RAM - Warning: 80-90% with minimal swap - Critical: >90% or active swapping I/O Wait Guidelines: - Normal: <10% iowait - Warning: 10-25% iowait - Critical: >25% sustained iowait Data Collection Best Practices 1. Consistent Timing: Use consistent intervals across tools 2. Sufficient Duration: Collect data long enough to identify patterns 3. Multiple Perspectives: Use different tools to validate findings 4. Historical Comparison: Compare current data with historical baselines Security Considerations ```bash Limit sensitive information exposure pidstat -u -C "apache|nginx" 5 10 # Only show web servers Use non-privileged monitoring when possible vmstat 5 10 # Works without root Secure log files chmod 640 /var/log/performance/*.log chown root:adm /var/log/performance/*.log ``` Conclusion Mastering CPU and memory monitoring with vmstat, mpstat, pidstat, and dstat provides essential skills for Linux system administration and performance optimization. Each tool offers unique capabilities: - vmstat: Comprehensive system overview with memory, CPU, and I/O statistics - mpstat: Detailed multi-processor CPU analysis and per-core monitoring - pidstat: Process-level resource consumption and performance tracking - dstat: Versatile, colorful, and flexible system monitoring with extensive plugin support Key Takeaways 1. Tool Selection: Choose the right tool for your specific monitoring needs 2. Interpretation Skills: Understanding metrics is more important than memorizing commands 3. Trend Analysis: Focus on patterns and trends rather than single data points 4. Comprehensive Monitoring: Use multiple tools together for complete system visibility 5. Proactive Monitoring: Regular monitoring prevents issues before they become critical Next Steps To further enhance your system monitoring capabilities: 1. Explore Advanced Features: Investigate tool-specific advanced options and plugins 2. Automate Monitoring: Create scripts and cron jobs for continuous monitoring 3. Learn Complementary Tools: Study iostat, sar, top, htop, and iotop 4. Practice Troubleshooting: Apply these tools to real-world performance issues 5. Study System Architecture: Deepen understanding of Linux kernel and hardware interaction By consistently applying these monitoring techniques and following best practices, you'll develop expertise in system performance analysis and become proficient at maintaining optimal Linux system performance. Remember that effective monitoring is an ongoing process that requires regular attention, proper tool selection, and continuous learning about system behavior patterns.