How to read system logs → journalctl

How to Read System Logs with journalctl: A Complete Guide Table of Contents 1. [Introduction](#introduction) 2. [Prerequisites](#prerequisites) 3. [Understanding systemd Journal](#understanding-systemd-journal) 4. [Basic journalctl Commands](#basic-journalctl-commands) 5. [Advanced Filtering and Search](#advanced-filtering-and-search) 6. [Time-Based Log Analysis](#time-based-log-analysis) 7. [Service-Specific Log Monitoring](#service-specific-log-monitoring) 8. [Log Formatting and Output Options](#log-formatting-and-output-options) 9. [Real-World Use Cases](#real-world-use-cases) 10. [Troubleshooting Common Issues](#troubleshooting-common-issues) 11. [Best Practices and Tips](#best-practices-and-tips) 12. [Conclusion](#conclusion) Introduction System logs are the backbone of Linux system administration, providing crucial insights into system behavior, errors, and performance issues. The `journalctl` command is the primary tool for reading and analyzing logs in modern Linux distributions that use systemd. This comprehensive guide will teach you everything you need to know about using journalctl to effectively read, filter, and analyze system logs. Whether you're a system administrator troubleshooting server issues, a developer debugging applications, or a Linux enthusiast wanting to understand your system better, mastering journalctl is essential. This article covers everything from basic log viewing to advanced filtering techniques, providing practical examples and real-world scenarios. Prerequisites Before diving into journalctl, ensure you have: - Linux Distribution: A systemd-based Linux distribution (most modern distributions including Ubuntu 16.04+, CentOS 7+, Fedora, Debian 8+, openSUSE) - User Privileges: Root access or sudo privileges for accessing system logs - Basic Command Line Knowledge: Familiarity with Linux terminal and basic commands - Text Editor: Basic understanding of text navigation and searching Checking systemd Availability Verify that your system uses systemd: ```bash systemctl --version ``` If systemd is running, you'll see version information. Otherwise, you may be using a different init system. Understanding systemd Journal What is the systemd Journal? The systemd journal is a centralized logging system that collects and stores log data from various sources: - Kernel messages: Hardware and driver information - System services: Service startup, shutdown, and operational messages - Applications: User-space application logs - Security events: Authentication and authorization logs Journal Storage Locations The journal stores logs in binary format in these locations: ```bash Runtime logs (lost on reboot) /run/log/journal/ Persistent logs (survive reboots) /var/log/journal/ ``` Advantages of systemd Journal - Centralized logging: All logs in one place - Structured data: Rich metadata for each log entry - Efficient storage: Binary format with compression - Powerful querying: Advanced filtering capabilities - Real-time monitoring: Live log streaming Basic journalctl Commands Viewing All Logs Display all available journal entries: ```bash journalctl ``` This command shows logs from oldest to newest, using a pager (usually `less`) for navigation. Recent Log Entries View the most recent log entries: ```bash Show last 20 entries journalctl -n 20 Show last 50 entries journalctl -n 50 Alternative syntax journalctl --lines=20 ``` Following Logs in Real-Time Monitor logs as they're generated (similar to `tail -f`): ```bash Follow all new log entries journalctl -f Follow with specific number of recent entries journalctl -f -n 30 ``` Reverse Order Display Show newest entries first: ```bash journalctl -r ``` No Pager Output Display logs directly to terminal without pager: ```bash journalctl --no-pager ``` Advanced Filtering and Search Priority Level Filtering Filter logs by severity level: ```bash Emergency level (0) - system unusable journalctl -p emerg Alert level (1) - immediate action required journalctl -p alert Critical level (2) - critical conditions journalctl -p crit Error level (3) - error conditions journalctl -p err Warning level (4) - warning conditions journalctl -p warning Notice level (5) - normal but significant journalctl -p notice Info level (6) - informational messages journalctl -p info Debug level (7) - debug messages journalctl -p debug ``` Filtering by Facility Filter logs by system facility: ```bash Kernel messages journalctl -f kern Mail system journalctl -f mail System daemons journalctl -f daemon Security/authorization messages journalctl -f auth ``` Text-Based Searching Search for specific text patterns: ```bash Search for specific text journalctl | grep "error" Case-insensitive search journalctl | grep -i "failed" Search with context lines journalctl | grep -A 5 -B 5 "connection refused" ``` Using journalctl's Built-in Search ```bash Search using journalctl's grep functionality journalctl -g "error" Case-insensitive search journalctl -g "(?i)failed" ``` Time-Based Log Analysis Viewing Logs Since Boot ```bash Current boot logs journalctl -b Previous boot logs journalctl -b -1 Specific boot (list available boots first) journalctl --list-boots journalctl -b 2 ``` Time Range Filtering Filter logs by specific time periods: ```bash Since specific date/time journalctl --since "2024-01-01 10:00:00" Until specific date/time journalctl --until "2024-01-01 18:00:00" Time range journalctl --since "2024-01-01" --until "2024-01-02" Relative time journalctl --since "1 hour ago" journalctl --since "30 minutes ago" journalctl --since "yesterday" journalctl --since "1 week ago" ``` Today's Logs ```bash Today's logs journalctl --since today Yesterday's logs journalctl --since yesterday --until today ``` Service-Specific Log Monitoring Viewing Service Logs Monitor specific systemd services: ```bash Apache web server journalctl -u apache2 SSH daemon journalctl -u ssh Network manager journalctl -u NetworkManager Multiple services journalctl -u apache2 -u mysql ``` Following Service Logs ```bash Follow Apache logs in real-time journalctl -u apache2 -f Follow with recent entries journalctl -u apache2 -f -n 50 ``` Service Logs with Time Filters ```bash Service logs since boot journalctl -u apache2 -b Service logs for specific time period journalctl -u apache2 --since "1 hour ago" ``` Log Formatting and Output Options Output Formats journalctl supports various output formats: ```bash Short format (default) journalctl -o short Verbose format with all fields journalctl -o verbose JSON format journalctl -o json JSON format (pretty printed) journalctl -o json-pretty Export format journalctl -o export Cat format (message only) journalctl -o cat ``` Custom Field Display Display specific fields: ```bash Show only timestamps and messages journalctl -o short-iso Show with microsecond precision journalctl -o short-precise Show with full timestamps journalctl -o short-full ``` Controlling Output Length ```bash Truncate long lines journalctl --no-full Show full content (default) journalctl --full Limit output to specific number of lines journalctl -n 100 --no-pager ``` Real-World Use Cases Troubleshooting Boot Issues When your system has boot problems: ```bash Check last boot logs journalctl -b -1 Look for error messages during boot journalctl -b -p err Check specific boot phase journalctl -b --since "2024-01-01 09:00:00" --until "2024-01-01 09:05:00" ``` Monitoring Web Server Issues For Apache/Nginx troubleshooting: ```bash Monitor Apache errors journalctl -u apache2 -p err -f Check recent Apache activity journalctl -u apache2 --since "1 hour ago" Look for specific error patterns journalctl -u apache2 | grep -i "404\|500\|error" ``` Database Problem Diagnosis For MySQL/PostgreSQL issues: ```bash MySQL service logs journalctl -u mysql -n 100 PostgreSQL startup issues journalctl -u postgresql -b Database connection errors journalctl -u mysql | grep -i "connection\|access denied" ``` Security Event Monitoring Monitor authentication and security events: ```bash SSH login attempts journalctl -u ssh --since today Failed authentication attempts journalctl | grep -i "failed\|authentication failure" Sudo usage journalctl | grep sudo ``` Network Connectivity Issues Diagnose network problems: ```bash NetworkManager logs journalctl -u NetworkManager -f DHCP client logs journalctl -u dhclient DNS resolution issues journalctl | grep -i "dns\|resolve" ``` Troubleshooting Common Issues Permission Denied Errors Problem: Cannot access journal logs Solution: ```bash Add user to systemd-journal group sudo usermod -a -G systemd-journal username Or use sudo sudo journalctl ``` Journal Size Issues Problem: Journal consuming too much disk space Solution: ```bash Check journal disk usage journalctl --disk-usage Clean old logs (keep last 2 weeks) sudo journalctl --vacuum-time=2weeks Limit journal size sudo journalctl --vacuum-size=500M Configure permanent limits sudo nano /etc/systemd/journald.conf Add: SystemMaxUse=500M ``` Missing Boot Logs Problem: Cannot see previous boot logs Solution: ```bash Enable persistent logging sudo mkdir -p /var/log/journal sudo systemctl restart systemd-journald Verify persistent storage journalctl --list-boots ``` Slow Journal Performance Problem: journalctl commands are slow Solution: ```bash Rebuild journal index sudo systemctl stop systemd-journald sudo rm /var/log/journal/*/system.journal~ sudo systemctl start systemd-journald Use more specific filters journalctl -u specific-service --since "1 hour ago" ``` Corrupted Journal Files Problem: Journal corruption errors Solution: ```bash Verify journal integrity journalctl --verify Force journal rotation sudo systemctl kill --kill-who=main --signal=SIGUSR2 systemd-journald ``` Best Practices and Tips Efficient Log Analysis 1. Use Specific Filters: Always narrow down your search with time ranges and service filters 2. Combine Multiple Filters: Use service, time, and priority filters together 3. Save Complex Queries: Create aliases for frequently used commands ```bash Add to ~/.bashrc alias apache-errors='journalctl -u apache2 -p err --since today' alias boot-errors='journalctl -b -p err' ``` Performance Optimization 1. Limit Output: Use `-n` to limit entries when possible 2. Use --no-pager: For scripts and automated processing 3. Specific Time Ranges: Avoid querying entire journal history Security Considerations 1. Protect Log Access: Ensure only authorized users can access logs 2. Monitor Log Tampering: Regularly verify journal integrity 3. Secure Log Storage: Consider encrypting log storage for sensitive systems Automation and Scripting Create monitoring scripts: ```bash #!/bin/bash Simple error monitoring script journalctl -p err --since "5 minutes ago" --no-pager | \ while read line; do echo "ERROR DETECTED: $line" # Send alert or take action done ``` Log Retention Management Configure appropriate log retention: ```bash Edit journald configuration sudo nano /etc/systemd/journald.conf Recommended settings SystemMaxUse=1G SystemKeepFree=500M MaxFileSec=1month MaxRetentionSec=6month ``` Integration with Other Tools Combine journalctl with other utilities: ```bash Export to syslog format journalctl -o syslog | logger -t imported-logs Send logs to remote server journalctl --since "1 hour ago" | ssh user@logserver 'cat >> /var/log/remote.log' Create log summaries journalctl -p err --since today --no-pager | wc -l ``` Conclusion Mastering journalctl is essential for effective Linux system administration and troubleshooting. This comprehensive guide has covered everything from basic log viewing to advanced filtering techniques, real-world use cases, and best practices. Key takeaways include: - Start Simple: Begin with basic commands and gradually incorporate advanced filters - Use Specific Filters: Always narrow your search with time, service, and priority filters - Monitor in Real-Time: Use `-f` flag for active troubleshooting - Automate Common Tasks: Create aliases and scripts for frequently used queries - Manage Log Storage: Configure appropriate retention policies to prevent disk space issues Regular practice with journalctl will significantly improve your ability to diagnose system issues, monitor application performance, and maintain system security. The centralized logging approach of systemd journal provides unprecedented visibility into your Linux system's operation. Continue exploring advanced features like journal forwarding, custom field filtering, and integration with monitoring tools to further enhance your log analysis capabilities. Remember that effective log analysis is both an art and a science – the more you practice, the more intuitive it becomes to identify patterns and diagnose issues quickly.