How to display kernel messages → dmesg - System Information & Monitoring Guide

How to Display Kernel Messages → dmesg Table of Contents - [Introduction](#introduction) - [Prerequisites](#prerequisites) - [Understanding Kernel Messages](#understanding-kernel-messages) - [Basic dmesg Usage](#basic-dmesg-usage) - [Advanced dmesg Options](#advanced-dmesg-options) - [Filtering and Searching Messages](#filtering-and-searching-messages) - [Real-World Examples](#real-world-examples) - [Troubleshooting Common Issues](#troubleshooting-common-issues) - [Best Practices](#best-practices) - [Alternative Methods](#alternative-methods) - [Conclusion](#conclusion) Introduction The `dmesg` command is one of the most essential tools in a Linux administrator's toolkit for diagnosing system issues, monitoring hardware events, and understanding kernel behavior. This comprehensive guide will teach you everything you need to know about displaying and interpreting kernel messages using dmesg, from basic usage to advanced filtering techniques. Kernel messages provide critical information about system boot processes, hardware detection, driver loading, error conditions, and various system events. Understanding how to effectively use dmesg can help you troubleshoot hardware problems, diagnose system crashes, monitor device connections, and maintain system health. By the end of this article, you'll be proficient in using dmesg to extract meaningful information from kernel logs, filter messages based on severity and time, and apply this knowledge to real-world system administration scenarios. Prerequisites Before diving into dmesg usage, ensure you have: - Linux System Access: Any Linux distribution with command-line access - Basic Terminal Knowledge: Familiarity with command-line interface - User Permissions: Regular user access (some operations may require sudo) - Text Processing Tools: Basic understanding of grep, less, and other text utilities System Requirements - Linux kernel 2.6 or later (most modern distributions) - Terminal emulator or SSH access - Sufficient privileges to read kernel ring buffer Understanding Kernel Messages What is the Kernel Ring Buffer? The kernel ring buffer is a data structure that stores kernel messages in memory. It has a fixed size and operates on a first-in, first-out basis, meaning older messages are overwritten when the buffer fills up. Key characteristics: - Fixed Size: Typically 16KB to 1MB depending on system configuration - Circular Buffer: New messages overwrite oldest when full - Memory-Based: Messages are stored in RAM, not on disk - Boot Messages: Contains messages from system startup Message Structure Kernel messages follow a specific format: ``` [timestamp] facility.level: message content ``` Example: ``` [ 0.000000] Linux version 5.4.0-74-generic (buildd@lcy01-amd64-029) [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-74-generic [ 1.234567] usb 1-1: new high-speed USB device number 2 using ehci-pci ``` Message Severity Levels Linux kernel uses eight severity levels (0-7): | Level | Name | Description | |-------|------|-------------| | 0 | EMERG | System is unusable | | 1 | ALERT | Action must be taken immediately | | 2 | CRIT | Critical conditions | | 3 | ERR | Error conditions | | 4 | WARNING | Warning conditions | | 5 | NOTICE | Normal but significant condition | | 6 | INFO | Informational messages | | 7 | DEBUG | Debug-level messages | Basic dmesg Usage Displaying All Messages The simplest way to view kernel messages: ```bash dmesg ``` This command displays all messages currently in the kernel ring buffer. The output can be overwhelming, so it's often piped through a pager: ```bash dmesg | less ``` Recent Messages To view the most recent messages, use the tail command: ```bash dmesg | tail dmesg | tail -20 # Last 20 messages ``` Human-Readable Timestamps By default, dmesg shows timestamps as seconds since boot. For human-readable format: ```bash dmesg -T ``` Example output: ``` [Mon Oct 25 14:30:45 2023] usb 1-1: USB disconnect, address 1 [Mon Oct 25 14:30:46 2023] usb 1-1: new high-speed USB device number 2 ``` Continuous Monitoring To monitor kernel messages in real-time: ```bash dmesg -w ``` This follows new messages as they appear, similar to `tail -f`. Advanced dmesg Options Facility-Based Filtering Filter messages by facility (subsystem): ```bash dmesg -f kern # Kernel messages dmesg -f user # User-space messages dmesg -f mail # Mail system messages dmesg -f daemon # System daemon messages ``` Level-Based Filtering Filter by severity level: ```bash dmesg -l err # Error messages only dmesg -l warn # Warning messages only dmesg -l info # Informational messages only ``` Combine multiple levels: ```bash dmesg -l err,warn,crit # Errors, warnings, and critical messages ``` Time-Based Filtering Show messages since a specific time: ```bash dmesg --since "2023-10-25 14:00:00" dmesg --since "1 hour ago" dmesg --since "10 minutes ago" ``` Show messages until a specific time: ```bash dmesg --until "2023-10-25 15:00:00" dmesg --until "30 minutes ago" ``` Clearing the Ring Buffer Clear all messages from the ring buffer (requires root privileges): ```bash sudo dmesg -c # Clear after displaying sudo dmesg -C # Clear without displaying ``` Raw Output Format Display raw format without facility/level decoding: ```bash dmesg -r ``` Kernel Log Levels Control which messages are displayed based on kernel log level: ```bash dmesg -n 1 # Only emergency messages dmesg -n 4 # Up to warning messages dmesg -n 7 # All messages including debug ``` Filtering and Searching Messages Using grep for Pattern Matching Combine dmesg with grep for powerful filtering: ```bash Find USB-related messages dmesg | grep -i usb Find error messages dmesg | grep -i error Find network interface messages dmesg | grep -E "(eth0|wlan0|enp)" Case-insensitive search for memory issues dmesg | grep -i "memory\|oom" ``` Advanced grep Patterns ```bash Find messages with context (5 lines before and after) dmesg | grep -C 5 "error" Find multiple patterns dmesg | grep -E "(failed|error|critical)" Exclude certain patterns dmesg | grep -v "audit" Count occurrences dmesg | grep -c "USB" ``` Using awk for Complex Filtering Extract specific fields or apply complex conditions: ```bash Show only timestamps and messages (remove log levels) dmesg | awk '{print $1, $2, $3}' Filter messages from last 60 seconds of boot dmesg | awk '$1 ~ /\[[ ]*[0-5][0-9]\.[0-9]+\]/ {print}' Show only messages with specific keywords dmesg | awk '/USB|PCI|SCSI/ {print}' ``` Combining Multiple Filters ```bash Recent error messages with human-readable timestamps dmesg -T -l err | tail -10 USB errors from the last hour dmesg -T --since "1 hour ago" | grep -i "usb.*error" Critical and error messages excluding audit dmesg -l err,crit | grep -v audit ``` Real-World Examples Hardware Troubleshooting Checking USB Device Issues: ```bash Monitor USB device connections dmesg -w | grep -i usb Check for USB errors dmesg | grep -i "usb.*$error\|failed\|disconnect$" Find USB device information dmesg | grep -A 5 -B 5 "new.*USB device" ``` Diagnosing Memory Problems: ```bash Check for out-of-memory conditions dmesg | grep -i "out of memory\|oom" Look for memory errors dmesg | grep -i "memory.error\|bad.memory" Check memory allocation issues dmesg | grep -E "(allocation|memory).*fail" ``` Network Interface Issues: ```bash Monitor network interface changes dmesg | grep -E "(eth[0-9]|wlan[0-9]|enp[0-9])" Check for network errors dmesg | grep -i "network.*$error\|timeout\|fail$" Monitor link status changes dmesg | grep -i "link.*$up\|down$" ``` Boot Process Analysis Analyzing Boot Sequence: ```bash Show boot messages chronologically dmesg -T | head -50 Check kernel version and command line dmesg | grep -E "(Linux version|Command line)" Monitor service startup dmesg | grep -i "systemd\|init" ``` Hardware Detection During Boot: ```bash CPU information dmesg | grep -i "cpu\|processor" Storage devices dmesg | grep -i "sd[a-z]\|nvme\|ata" Graphics hardware dmesg | grep -i "gpu\|graphics\|drm" ``` Security Monitoring Checking Security Events: ```bash SELinux denials dmesg | grep -i "selinux.*denied" Firewall messages dmesg | grep -i "iptables\|netfilter" Failed authentication attempts dmesg | grep -i "authentication.*fail" ``` Performance Issues I/O Problems: ```bash Disk I/O errors dmesg | grep -i "i/o error\|sector.*error" Filesystem errors dmesg | grep -i "filesystem.error\|ext[234].error" Storage timeouts dmesg | grep -i "timeout.*$scsi\|ata\|nvme$" ``` Troubleshooting Common Issues Empty dmesg Output Problem: `dmesg` returns no output or "dmesg: read kernel buffer failed: Operation not permitted" Solutions: ```bash Check if you need elevated privileges sudo dmesg Verify kernel ring buffer size cat /proc/sys/kernel/dmesg_restrict Check if messages exist in alternative locations journalctl -k # systemd systems cat /var/log/kern.log # if available ``` Missing Timestamps Problem: Timestamps appear as seconds since boot instead of human-readable format Solutions: ```bash Use -T flag for human-readable timestamps dmesg -T If -T doesn't work, check system configuration cat /proc/uptime # Verify system uptime date # Verify system clock ``` Buffer Overflow Problem: Important messages are lost due to ring buffer overflow Solutions: ```bash Increase ring buffer size (requires reboot) Add to kernel command line: log_buf_len=1M Use persistent logging sudo systemctl enable systemd-journald journalctl -k # View kernel messages from journal Monitor continuously for important events dmesg -w | grep -E "(error|critical|alert)" >> /tmp/kernel_errors.log ``` Permission Denied Problem: Cannot read kernel messages due to restricted access Solutions: ```bash Check dmesg restriction setting cat /proc/sys/kernel/dmesg_restrict Temporarily allow access (as root) echo 0 | sudo tee /proc/sys/kernel/dmesg_restrict Permanently allow access echo "kernel.dmesg_restrict = 0" | sudo tee -a /etc/sysctl.conf ``` Garbled Output Problem: dmesg output contains control characters or appears corrupted Solutions: ```bash Use raw output format dmesg -r Filter control characters dmesg | tr -d '\000-\010\013\014\016-\037\177-\377' Use alternative display methods dmesg | cat -v # Show non-printing characters dmesg | less -R # Handle color codes properly ``` Best Practices Regular Monitoring Establish Monitoring Routines: ```bash Create daily kernel message summary #!/bin/bash echo "=== Kernel Messages Summary - $(date) ===" >> /var/log/daily_kernel_summary.log dmesg -l err,crit,alert,emerg >> /var/log/daily_kernel_summary.log echo "" >> /var/log/daily_kernel_summary.log ``` Automated Error Detection: ```bash Script to alert on critical messages #!/bin/bash CRITICAL_MSGS=$(dmesg -l emerg,alert,crit | wc -l) if [ $CRITICAL_MSGS -gt 0 ]; then echo "Critical kernel messages detected!" | mail -s "Kernel Alert" admin@example.com dmesg -l emerg,alert,crit | mail -s "Critical Messages" admin@example.com fi ``` Log Preservation Save Important Messages: ```bash Backup current kernel messages dmesg -T > /var/log/dmesg.$(date +%Y%m%d_%H%M%S) Create rotating kernel message logs dmesg -T | logger -t kernel-messages ``` Integration with System Logging: ```bash Ensure kernel messages are logged persistently Check rsyslog configuration grep -r "kern\." /etc/rsyslog* Verify systemd journal is capturing kernel messages journalctl -k --no-pager | tail ``` Performance Considerations Efficient Message Processing: ```bash Use specific filters to reduce processing time dmesg -l err,warn --since "1 hour ago" | grep -E "(usb|network|disk)" Avoid repeatedly calling dmesg in scripts Store output in variable for multiple operations DMESG_OUTPUT=$(dmesg -T) echo "$DMESG_OUTPUT" | grep error echo "$DMESG_OUTPUT" | grep warning ``` Documentation and Analysis Create Message Catalogs: ```bash Document recurring messages and their meanings dmesg | sort | uniq -c | sort -nr > /tmp/message_frequency.txt Categorize messages by subsystem for subsys in usb pci acpi thermal; do echo "=== $subsys messages ===" >> /tmp/subsystem_messages.txt dmesg | grep -i $subsys >> /tmp/subsystem_messages.txt echo "" >> /tmp/subsystem_messages.txt done ``` Alternative Methods Using journalctl On systemd-based systems, journalctl provides an alternative to dmesg: ```bash View kernel messages journalctl -k Follow kernel messages in real-time journalctl -k -f Kernel messages since last boot journalctl -k -b Kernel messages with specific priority journalctl -k -p err ``` Reading from /proc/kmsg Direct access to kernel messages: ```bash Read kernel messages directly (requires root) sudo cat /proc/kmsg Monitor new messages sudo tail -f /proc/kmsg ``` Warning: Reading from `/proc/kmsg` consumes messages, making them unavailable to other processes. Log Files Traditional log files may contain kernel messages: ```bash Check common log locations tail -f /var/log/messages tail -f /var/log/kern.log tail -f /var/log/syslog ``` Using sysctl Configure kernel message behavior: ```bash View current settings sysctl kernel.printk sysctl kernel.dmesg_restrict Modify message levels sudo sysctl kernel.printk="4 4 1 7" ``` Conclusion The `dmesg` command is an indispensable tool for Linux system administration, providing crucial insights into kernel behavior, hardware status, and system health. Throughout this comprehensive guide, we've explored everything from basic usage to advanced filtering techniques, real-world troubleshooting scenarios, and best practices for effective kernel message analysis. Key takeaways from this guide include: - Fundamental Understanding: dmesg displays messages from the kernel ring buffer, providing real-time and historical information about system events - Versatile Filtering: Advanced options like time-based filtering, severity levels, and facility-based selection allow precise message extraction - Practical Applications: From hardware troubleshooting to security monitoring, dmesg serves multiple critical functions in system administration - Integration Capabilities: Combining dmesg with other tools like grep, awk, and journalctl creates powerful diagnostic workflows Next Steps To further enhance your kernel message analysis skills: 1. Practice Regular Monitoring: Implement daily dmesg checks as part of your system maintenance routine 2. Develop Custom Scripts: Create automated monitoring solutions tailored to your specific environment 3. Study Message Patterns: Build familiarity with common message types in your systems 4. Integrate with Monitoring Tools: Incorporate dmesg analysis into broader system monitoring solutions 5. Explore Related Tools: Learn complementary tools like journalctl, systemctl, and various log analysis utilities Remember that effective use of dmesg comes with experience. Regular practice with different filtering options, combined with understanding your specific hardware and software environment, will make you proficient in quickly identifying and resolving system issues through kernel message analysis. The kernel messages displayed by dmesg are often your first and most reliable source of information when diagnosing system problems. Master this tool, and you'll have a powerful ally in maintaining stable, well-functioning Linux systems.