How to display system statistics → vmstat

How to Display System Statistics → vmstat Table of Contents 1. [Introduction](#introduction) 2. [Prerequisites](#prerequisites) 3. [Understanding vmstat](#understanding-vmstat) 4. [Basic vmstat Usage](#basic-vmstat-usage) 5. [vmstat Output Explained](#vmstat-output-explained) 6. [Advanced vmstat Options](#advanced-vmstat-options) 7. [Practical Examples and Use Cases](#practical-examples-and-use-cases) 8. [Monitoring Strategies](#monitoring-strategies) 9. [Common Issues and Troubleshooting](#common-issues-and-troubleshooting) 10. [Best Practices](#best-practices) 11. [Alternative Tools](#alternative-tools) 12. [Conclusion](#conclusion) Introduction The `vmstat` (Virtual Memory Statistics) command is one of the most fundamental and powerful tools available to Linux system administrators for monitoring system performance. This versatile utility provides real-time insights into virtual memory, processes, CPU activity, and I/O statistics, making it an essential component of any system monitoring toolkit. In this comprehensive guide, you'll learn how to effectively use vmstat to monitor your Linux systems, interpret its output, and leverage its various options for different monitoring scenarios. Whether you're troubleshooting performance issues, capacity planning, or maintaining system health, vmstat provides the critical metrics you need to make informed decisions. By the end of this article, you'll have a thorough understanding of vmstat's capabilities and be able to implement effective monitoring strategies using this powerful command-line tool. Prerequisites Before diving into vmstat usage, ensure you have: - Linux System Access: A Linux-based system (Ubuntu, CentOS, Red Hat, Debian, etc.) - Command Line Knowledge: Basic familiarity with Linux command line interface - User Permissions: Standard user access (root privileges not required for basic usage) - Terminal Access: SSH access or direct terminal access to the system - Basic System Knowledge: Understanding of concepts like CPU, memory, and processes System Requirements vmstat is typically pre-installed on most Linux distributions as part of the `procps` or `procps-ng` package. To verify installation: ```bash Check if vmstat is available which vmstat Check vmstat version vmstat --version ``` If vmstat is not installed, you can install it using your distribution's package manager: ```bash Ubuntu/Debian sudo apt-get install procps CentOS/RHEL/Fedora sudo yum install procps-ng or for newer versions sudo dnf install procps-ng ``` Understanding vmstat What is vmstat? vmstat reports information about processes, memory, paging, block I/O, traps, disks, and CPU activity. It's particularly valuable because it provides a snapshot of system performance metrics in a single command, making it ideal for quick system health checks and ongoing monitoring. Key Metrics Provided vmstat displays several categories of system information: - Process Information: Running and blocked processes - Memory Statistics: Free, used, buffer, and cache memory - Swap Activity: Swap in and swap out operations - I/O Statistics: Block input and output operations - System Activity: Interrupts and context switches per second - CPU Utilization: User, system, idle, and wait time percentages When to Use vmstat vmstat is particularly useful for: - Performance Troubleshooting: Identifying bottlenecks in CPU, memory, or I/O - Capacity Planning: Understanding resource utilization trends - System Monitoring: Regular health checks and baseline establishment - Problem Diagnosis: Investigating high load, memory issues, or I/O problems - Baseline Creation: Establishing normal operating parameters Basic vmstat Usage Simple vmstat Command The most basic vmstat usage displays current system statistics: ```bash vmstat ``` This command shows a single snapshot of system statistics since boot time. The output provides averages calculated from system startup. Continuous Monitoring For ongoing monitoring, specify an interval (in seconds): ```bash Update every 2 seconds vmstat 2 Update every 5 seconds vmstat 5 ``` Limited Iterations To limit the number of reports, specify both interval and count: ```bash Update every 2 seconds, 10 times total vmstat 2 10 Update every 1 second, 5 times total vmstat 1 5 ``` Example Output ```bash $ vmstat 2 5 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 1847532 92576 1456789 0 0 12 8 45 89 2 1 97 0 0 0 0 0 1847234 92576 1456823 0 0 0 0 78 156 1 1 98 0 0 2 0 0 1846987 92576 1456845 0 0 0 24 95 178 3 2 95 0 0 1 0 0 1846754 92576 1456867 0 0 0 0 67 134 1 1 98 0 0 0 0 0 1846521 92576 1456889 0 0 0 8 89 167 2 1 97 0 0 ``` vmstat Output Explained Understanding vmstat output is crucial for effective system monitoring. Let's break down each column: Process Statistics (procs) - r (running): Number of processes waiting for CPU time - b (blocked): Number of processes in uninterruptible sleep (usually I/O) ```bash High 'r' values indicate CPU bottleneck High 'b' values suggest I/O bottleneck ``` Memory Statistics (memory) All memory values are displayed in kilobytes by default: - swpd: Virtual memory used (KB) - free: Idle memory (KB) - buff: Memory used as buffers (KB) - cache: Memory used as cache (KB) Swap Statistics (swap) - si (swap in): Memory swapped in from disk (KB/s) - so (swap out): Memory swapped to disk (KB/s) ```bash Non-zero si/so values indicate memory pressure ``` I/O Statistics (io) - bi (blocks in): Blocks received from block device (blocks/s) - bo (blocks out): Blocks sent to block device (blocks/s) System Statistics (system) - in (interrupts): Interrupts per second, including clock - cs (context switches): Context switches per second CPU Statistics (cpu) Percentages of total CPU time: - us (user): Time spent running non-kernel code - sy (system): Time spent running kernel code - id (idle): Time spent idle - wa (wait): Time spent waiting for I/O - st (stolen): Time stolen from virtual machine Advanced vmstat Options Display Units Change the display units for memory statistics: ```bash Display in megabytes vmstat -S M Display in kilobytes (default) vmstat -S k Display in bytes vmstat -S b ``` Active vs Inactive Memory Show active and inactive memory statistics: ```bash vmstat -a ``` Example output: ```bash $ vmstat -a procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free inact active si so bi bo in cs us sy id wa st 1 0 0 1847532 456789 1234567 0 0 12 8 45 89 2 1 97 0 0 ``` Fork Statistics Display fork statistics: ```bash vmstat -f ``` This shows the total number of forks since boot: ```bash $ vmstat -f 2847691 forks ``` Slab Information Display slab allocator statistics: ```bash vmstat -m ``` Disk Statistics Show disk statistics: ```bash vmstat -d ``` Example output: ```bash $ vmstat -d disk- ------------reads------------ ------------writes----------- -----IO------ total merged sectors ms total merged sectors ms cur sec sda 123456 1234 9876543 45678 67890 5678 3456789 23456 0 78 ``` Partition Statistics Display statistics for specific partitions: ```bash vmstat -p /dev/sda1 ``` Wide Output Format Use wider output format for better readability: ```bash vmstat -w ``` Practical Examples and Use Cases Example 1: CPU Performance Analysis Monitor CPU utilization patterns: ```bash Monitor every second for 60 iterations vmstat 1 60 ``` Analysis Points: - High us values: CPU-intensive applications - High sy values: Kernel overhead or system calls - High wa values: I/O bottlenecks affecting CPU - Low id values: CPU saturation Example 2: Memory Pressure Detection Identify memory-related issues: ```bash Monitor with memory unit display vmstat -S M 2 ``` Key Indicators: - si/so > 0: Active swapping indicates memory pressure - free consistently low: Potential memory shortage - cache shrinking: System reclaiming cache for applications Example 3: I/O Performance Monitoring Track disk I/O patterns: ```bash Combine vmstat with disk statistics vmstat -d 5 ``` Analysis Focus: - High bi/bo values: Heavy disk activity - b column > 0: Processes waiting for I/O - wa percentage: CPU time lost to I/O wait Example 4: System Load Investigation Comprehensive system analysis: ```bash #!/bin/bash System monitoring script echo "System Performance Report - $(date)" echo "==================================" echo echo "Current System Statistics:" vmstat 1 1 echo echo "Memory Details:" vmstat -S M -a 1 1 echo echo "Fork Activity:" vmstat -f ``` Example 5: Performance Baseline Creation Establish system performance baselines: ```bash Create hourly performance snapshots for hour in {1..24}; do echo "Hour $hour: $(date)" >> baseline.log vmstat 60 60 >> baseline.log sleep 3600 done ``` Monitoring Strategies Real-time Monitoring For immediate performance analysis: ```bash Quick system health check vmstat 1 10 Detailed monitoring session vmstat -w -S M 2 ``` Automated Monitoring Create monitoring scripts for continuous observation: ```bash #!/bin/bash performance_monitor.sh LOG_FILE="/var/log/vmstat_monitor.log" INTERVAL=300 # 5 minutes COUNT=288 # 24 hours worth of data while true; do echo "$(date): Starting monitoring cycle" >> $LOG_FILE vmstat $INTERVAL $COUNT >> $LOG_FILE sleep 60 done ``` Threshold-based Alerting Implement basic alerting based on vmstat metrics: ```bash #!/bin/bash vmstat_alert.sh Get current statistics STATS=$(vmstat 1 2 | tail -1) CPU_IDLE=$(echo $STATS | awk '{print $15}') MEMORY_FREE=$(echo $STATS | awk '{print $4}') SWAP_USED=$(echo $STATS | awk '{print $3}') Define thresholds CPU_THRESHOLD=10 MEMORY_THRESHOLD=100000 SWAP_THRESHOLD=1000000 Check thresholds and alert if [ $CPU_IDLE -lt $CPU_THRESHOLD ]; then echo "ALERT: High CPU usage - Idle: ${CPU_IDLE}%" fi if [ $MEMORY_FREE -lt $MEMORY_THRESHOLD ]; then echo "ALERT: Low memory - Free: ${MEMORY_FREE}KB" fi if [ $SWAP_USED -gt $SWAP_THRESHOLD ]; then echo "ALERT: High swap usage - Used: ${SWAP_USED}KB" fi ``` Historical Analysis Analyze historical performance data: ```bash Extract specific metrics from logs grep "$(date '+%Y-%m-%d')" /var/log/vmstat.log | \ awk '{sum+=$15; count++} END {print "Average CPU idle:", sum/count "%"}' ``` Common Issues and Troubleshooting Issue 1: vmstat Command Not Found Problem: `vmstat: command not found` Solution: ```bash Install procps package Ubuntu/Debian sudo apt-get update && sudo apt-get install procps CentOS/RHEL sudo yum install procps-ng ``` Issue 2: Permission Denied Errors Problem: Cannot access certain system files Solution: ```bash Most vmstat functions work with regular user privileges For disk statistics, you might need elevated privileges sudo vmstat -d ``` Issue 3: Misleading First Line Statistics Problem: First vmstat output line shows unusual values Explanation: The first line shows averages since boot, not current activity. Solution: ```bash Skip first line or use interval mode vmstat 1 5 | tail -n +4 ``` Issue 4: High Memory Usage Confusion Problem: System shows low free memory but performs well Explanation: Linux uses available memory for caching. Analysis: ```bash Check available memory (free + cache + buffers) vmstat -S M Available memory = free + buff + cache ``` Issue 5: Interpreting Swap Activity Problem: Occasional swap activity causes concern Solution: Distinguish between normal and problematic swapping: ```bash Monitor swap trends vmstat 5 12 Consistent si/so values indicate memory pressure Occasional activity may be normal cleanup ``` Issue 6: CPU Wait Time Interpretation Problem: High wa (wait) values misunderstood as CPU issues Explanation: High wa indicates I/O bottlenecks, not CPU problems. Investigation: ```bash Correlate with I/O statistics vmstat -d Check for high disk utilization iostat -x 1 ``` Best Practices Monitoring Frequency Choose appropriate monitoring intervals: - Real-time troubleshooting: 1-2 seconds - Regular monitoring: 5-30 seconds - Long-term analysis: 1-5 minutes - Historical logging: 5-15 minutes Data Collection Implement systematic data collection: ```bash Structured logging approach vmstat -t 300 >> /var/log/vmstat-$(date +%Y%m%d).log ``` Baseline Establishment Create performance baselines during different periods: ```bash Peak hours baseline vmstat 60 60 > baseline_peak.log Off-hours baseline vmstat 300 72 > baseline_offhours.log ``` Correlation with Other Tools Combine vmstat with complementary tools: ```bash Comprehensive system analysis { echo "=== VMSTAT ===" vmstat 1 5 echo "=== IOSTAT ===" iostat -x 1 5 echo "=== TOP ===" top -b -n 1 | head -20 } > system_report.txt ``` Automation and Scripting Create reusable monitoring scripts: ```bash #!/bin/bash comprehensive_monitor.sh DATE=$(date +%Y%m%d_%H%M%S) REPORT_DIR="/var/log/performance" mkdir -p $REPORT_DIR System overview vmstat > $REPORT_DIR/vmstat_$DATE.log Extended monitoring vmstat 5 720 > $REPORT_DIR/vmstat_extended_$DATE.log & echo "Monitoring started. PID: $!" ``` Performance Tuning Insights Use vmstat data for system optimization: - High swap activity: Increase RAM or optimize applications - High CPU wait: Investigate disk I/O bottlenecks - Low CPU idle: Consider CPU upgrade or load balancing - High context switches: Investigate application threading Alternative Tools While vmstat is excellent for general system monitoring, consider these alternatives for specific use cases: htop Interactive process viewer with real-time updates: ```bash htop ``` iostat Detailed I/O statistics: ```bash iostat -x 1 ``` sar (System Activity Reporter) Comprehensive system activity reporting: ```bash sar -u 1 10 # CPU utilization sar -r 1 10 # Memory utilization ``` top Process-focused system monitoring: ```bash top -d 1 ``` nmon Comprehensive performance monitoring: ```bash nmon ``` Conclusion The vmstat command is an indispensable tool for Linux system administrators and performance analysts. Its ability to provide comprehensive system statistics in a simple, readable format makes it perfect for both quick health checks and detailed performance analysis. Throughout this guide, we've covered: - Basic Usage: Simple commands for immediate system insights - Advanced Options: Specialized flags for detailed analysis - Output Interpretation: Understanding what the numbers mean - Practical Applications: Real-world monitoring scenarios - Troubleshooting: Common issues and their solutions - Best Practices: Professional monitoring strategies Key Takeaways 1. Start Simple: Use basic vmstat commands for initial system assessment 2. Monitor Continuously: Implement regular monitoring for trend analysis 3. Understand Context: Interpret metrics in relation to system workload 4. Combine Tools: Use vmstat alongside other monitoring utilities 5. Automate Collection: Create scripts for consistent data gathering 6. Establish Baselines: Know your system's normal operating parameters Next Steps To further enhance your system monitoring capabilities: 1. Practice Regularly: Use vmstat in your daily system administration tasks 2. Create Monitoring Scripts: Develop automated monitoring solutions 3. Learn Complementary Tools: Explore iostat, sar, and other performance utilities 4. Implement Alerting: Set up threshold-based notifications 5. Study Performance Patterns: Analyze historical data for optimization opportunities With vmstat in your toolkit, you're well-equipped to monitor, analyze, and optimize Linux system performance effectively. Regular use of this powerful command will help you maintain healthy systems and quickly identify performance issues before they impact users. Remember that effective system monitoring is not just about collecting data—it's about understanding what the data tells you about your system's health and performance. vmstat provides the foundation for this understanding, giving you the insights needed to make informed decisions about system management and optimization.