How to view logs with journalctl

How to View Logs with journalctl Table of Contents 1. [Introduction](#introduction) 2. [Prerequisites](#prerequisites) 3. [Understanding journalctl and systemd Journal](#understanding-journalctl-and-systemd-journal) 4. [Basic journalctl Commands](#basic-journalctl-commands) 5. [Filtering and Searching Logs](#filtering-and-searching-logs) 6. [Time-Based Log Filtering](#time-based-log-filtering) 7. [Service-Specific Log Viewing](#service-specific-log-viewing) 8. [Advanced journalctl Features](#advanced-journalctl-features) 9. [Log Management and Maintenance](#log-management-and-maintenance) 10. [Common Use Cases and Examples](#common-use-cases-and-examples) 11. [Troubleshooting Common Issues](#troubleshooting-common-issues) 12. [Best Practices and Tips](#best-practices-and-tips) 13. [Conclusion](#conclusion) Introduction The `journalctl` command is a powerful tool for viewing and managing system logs on Linux distributions that use systemd. As the primary interface to the systemd journal, journalctl provides administrators and users with comprehensive access to system logs, service messages, kernel logs, and application output in a unified, searchable format. This comprehensive guide will teach you everything you need to know about using journalctl effectively, from basic log viewing to advanced filtering techniques. Whether you're a system administrator troubleshooting server issues, a developer debugging applications, or a Linux enthusiast learning system management, mastering journalctl is essential for effective log analysis and system monitoring. By the end of this article, you'll understand how to navigate system logs efficiently, filter information based on various criteria, troubleshoot common problems, and implement best practices for log management using journalctl. Prerequisites Before diving into journalctl usage, ensure you have: - Linux Distribution with systemd: Most modern Linux distributions (Ubuntu 16.04+, CentOS 7+, Fedora, Debian 8+, RHEL 7+) use systemd - Basic Command Line Knowledge: Familiarity with terminal usage and basic Linux commands - User Permissions: Root access or sudo privileges for viewing system-wide logs - Terminal Access: SSH access or local terminal session Checking systemd Availability Verify that your system uses systemd: ```bash Check if systemd is running systemctl --version Verify systemd is the init system ps -p 1 -o comm= ``` If the output shows "systemd," you're ready to proceed with journalctl. Understanding journalctl and systemd Journal What is systemd Journal? The systemd journal is a centralized logging system that collects and stores log data from various sources including: - Kernel messages: System-level events and errors - System services: Service startup, shutdown, and operational messages - User applications: Application logs and error messages - Security events: Authentication and authorization logs - Hardware events: Device-related messages Key Features of journalctl - Binary Format: Logs are stored in a structured binary format for efficient storage and retrieval - Indexing: Built-in indexing enables fast searching and filtering - Metadata: Each log entry includes extensive metadata (timestamps, process IDs, service names, etc.) - Rotation: Automatic log rotation based on size and time limits - Security: Access control and integrity verification Journal Storage Locations The systemd journal stores logs in: - Runtime logs: `/run/log/journal/` (temporary, lost on reboot) - Persistent logs: `/var/log/journal/` (permanent storage) Basic journalctl Commands Viewing All Logs The most basic journalctl command displays all available log entries: ```bash View all logs (newest first) journalctl View logs in chronological order (oldest first) journalctl --reverse ``` Paging Through Logs By default, journalctl uses a pager (usually `less`) for navigation: - Space: Next page - b: Previous page - q: Quit - /search_term: Search for text - G: Go to end - g: Go to beginning Following Logs in Real-Time Monitor logs as they're generated: ```bash Follow new log entries (similar to tail -f) journalctl -f Follow logs for the last 10 lines journalctl -f -n 10 ``` Limiting Output Control the number of log entries displayed: ```bash Show last 20 log entries journalctl -n 20 Show last 100 entries journalctl --lines=100 ``` Filtering and Searching Logs Priority-Based Filtering Filter logs by severity level using priority options: ```bash Show only error messages and above journalctl -p err Show warning messages and above journalctl -p warning Show info messages and above journalctl -p info ``` Priority levels (from highest to lowest): - emerg (0): Emergency messages - alert (1): Alert messages - crit (2): Critical messages - err (3): Error messages - warning (4): Warning messages - notice (5): Notice messages - info (6): Info messages - debug (7): Debug messages Filtering by Process ID View logs from specific processes: ```bash Filter by process ID journalctl _PID=1234 Filter by multiple PIDs journalctl _PID=1234 _PID=5678 ``` Filtering by User View logs related to specific users: ```bash Show logs for specific user ID journalctl _UID=1000 Show logs for current user journalctl _UID=$(id -u) ``` Text-Based Searching Search for specific text patterns: ```bash Search for specific text (case-insensitive) journalctl | grep -i "error" Use journalctl's built-in grep functionality journalctl --grep="failed" Case-sensitive search journalctl --case-sensitive --grep="Error" ``` Time-Based Log Filtering Viewing Logs Since Boot ```bash Show logs since current boot journalctl -b Show logs from previous boot journalctl -b -1 Show logs from specific boot (list boots first) journalctl --list-boots journalctl -b 2 ``` Date and Time Filtering ```bash Show logs since specific date journalctl --since "2024-01-01" Show logs until specific date journalctl --until "2024-01-31" Show logs for specific time range journalctl --since "2024-01-01 10:00:00" --until "2024-01-01 11:00:00" Show logs from last hour journalctl --since "1 hour ago" Show logs from last 30 minutes journalctl --since "30 min ago" Show logs from yesterday journalctl --since yesterday --until today ``` Relative Time Examples ```bash Last 24 hours journalctl --since "24 hours ago" Last week journalctl --since "1 week ago" Last month journalctl --since "1 month ago" Between 2 and 1 hours ago journalctl --since "2 hours ago" --until "1 hour ago" ``` Service-Specific Log Viewing Viewing Service Logs Monitor specific systemd services: ```bash View logs for specific service journalctl -u nginx.service Follow logs for a service journalctl -u apache2.service -f View logs for multiple services journalctl -u nginx.service -u mysql.service Show only today's logs for a service journalctl -u sshd.service --since today ``` Kernel Message Filtering View kernel-related logs: ```bash Show kernel messages only journalctl -k Show kernel messages since boot journalctl -k -b Follow kernel messages journalctl -k -f ``` System vs User Logs ```bash Show system-wide logs only journalctl --system Show user session logs only journalctl --user Show logs for specific user session journalctl --user -u user@1000.service ``` Advanced journalctl Features JSON Output Format Export logs in JSON format for processing: ```bash Output in JSON format journalctl -o json Pretty-printed JSON journalctl -o json-pretty JSON with one entry per line journalctl -o json-seq ``` Different Output Formats ```bash Short format (default) journalctl -o short Verbose format with all metadata journalctl -o verbose Only the message content journalctl -o cat Export format for backup/transfer journalctl -o export ``` Field-Based Filtering Filter using specific journal fields: ```bash Filter by systemd unit journalctl _SYSTEMD_UNIT=nginx.service Filter by executable path journalctl _EXE=/usr/sbin/sshd Filter by command name journalctl _COMM=systemd Show available fields journalctl -N ``` Combining Multiple Filters ```bash Multiple conditions (AND logic) journalctl _SYSTEMD_UNIT=nginx.service -p err Time range with service filtering journalctl -u mysql.service --since "1 hour ago" -p warning Complex filtering example journalctl _SYSTEMD_UNIT=sshd.service --since "2024-01-01" --until "2024-01-02" -p info ``` Log Management and Maintenance Checking Journal Size Monitor journal storage usage: ```bash Show current journal disk usage journalctl --disk-usage Show detailed journal file information ls -lh /var/log/journal/*/ Check journal verification journalctl --verify ``` Cleaning Up Logs Manage journal storage space: ```bash Remove logs older than 2 weeks journalctl --vacuum-time=2weeks Keep only 1GB of logs journalctl --vacuum-size=1G Keep only 100 log files journalctl --vacuum-files=100 Rotate journal files systemctl kill --kill-who=main --signal=SIGUSR2 systemd-journald.service ``` Configuring Journal Settings Edit `/etc/systemd/journald.conf` for persistent settings: ```ini [Journal] Maximum disk space for journal SystemMaxUse=1G Maximum file size SystemMaxFileSize=100M How long to keep logs MaxRetentionSec=1month Storage location Storage=persistent ``` Apply configuration changes: ```bash sudo systemctl restart systemd-journald ``` Common Use Cases and Examples Troubleshooting Boot Issues ```bash View logs from failed boot journalctl --list-boots journalctl -b -1 -p err Check for boot-related errors journalctl -b -u systemd-* --since "10 min ago" Monitor boot process journalctl -f -u systemd-logind.service ``` Debugging Network Services ```bash SSH connection issues journalctl -u sshd.service --since "1 hour ago" | grep -i "failed\|error" Web server problems journalctl -u nginx.service -u apache2.service -p warning --since today Network interface issues journalctl -k | grep -i "network\|eth0\|wlan0" ``` Security Monitoring ```bash Failed login attempts journalctl _SYSTEMD_UNIT=sshd.service | grep "Failed password" Sudo usage monitoring journalctl _COMM=sudo --since "1 day ago" Authentication logs journalctl _SYSTEMD_UNIT=systemd-logind.service -p info ``` Application Debugging ```bash Database service issues journalctl -u postgresql.service -u mysql.service --since "30 min ago" -p err Web application errors journalctl -u php7.4-fpm.service --since "1 hour ago" | grep -i "error\|fatal" Container runtime logs journalctl -u docker.service -u containerd.service -f ``` Troubleshooting Common Issues Issue: Permission Denied Problem: Cannot view logs due to permission restrictions. Solution: ```bash Add user to systemd-journal group sudo usermod -a -G systemd-journal $USER Or use sudo for one-time access sudo journalctl Check current permissions ls -la /var/log/journal/ ``` Issue: No Logs Showing Problem: journalctl returns no output or very few entries. Solutions: ```bash Check if journal service is running systemctl status systemd-journald Verify journal storage configuration sudo journalctl --verify Check available boots journalctl --list-boots Force journal sync sudo systemctl kill --signal=SIGUSR1 systemd-journald.service ``` Issue: Logs Taking Too Much Space Problem: Journal files consuming excessive disk space. Solutions: ```bash Check current usage journalctl --disk-usage Clean old logs sudo journalctl --vacuum-time=1month sudo journalctl --vacuum-size=500M Configure automatic limits in /etc/systemd/journald.conf SystemMaxUse=1G SystemMaxFileSize=100M ``` Issue: Slow Log Queries Problem: journalctl commands taking too long to execute. Solutions: ```bash Use more specific filters journalctl -u specific.service --since "1 hour ago" Limit output size journalctl -n 100 Use priority filtering journalctl -p warning Check journal integrity sudo journalctl --verify ``` Issue: Missing Boot Logs Problem: Cannot see logs from previous boots. Solutions: ```bash Ensure persistent storage sudo mkdir -p /var/log/journal sudo systemctl restart systemd-journald Check storage configuration grep Storage /etc/systemd/journald.conf Set persistent storage echo "Storage=persistent" | sudo tee -a /etc/systemd/journald.conf ``` Best Practices and Tips Performance Optimization 1. Use Specific Filters: Always filter logs to reduce processing time: ```bash # Good: Specific service and time range journalctl -u nginx.service --since "1 hour ago" # Avoid: Viewing all logs without filters journalctl | grep nginx ``` 2. Limit Output Size: Use `-n` option to limit entries: ```bash journalctl -n 50 -u mysql.service ``` 3. Use Appropriate Priority Levels: Filter by severity to focus on important messages: ```bash journalctl -p err --since today ``` Security Considerations 1. Protect Log Access: Ensure proper permissions on journal files 2. Regular Monitoring: Set up automated monitoring for critical errors 3. Log Retention: Balance storage space with compliance requirements 4. Audit Trail: Monitor access to sensitive logs Automation and Scripting Create useful scripts for common tasks: ```bash #!/bin/bash check-errors.sh - Daily error report echo "=== Daily Error Report ===" echo "Date: $(date)" echo "=== Critical Errors ===" journalctl -p crit --since "24 hours ago" --no-pager echo "=== Service Failures ===" journalctl _SYSTEMD_UNIT=*.service -p err --since "24 hours ago" --no-pager ``` Monitoring Best Practices 1. Regular Health Checks: Monitor journal service status 2. Storage Management: Implement automated cleanup 3. Alert Configuration: Set up alerts for critical errors 4. Documentation: Keep records of common issues and solutions Integration with Other Tools Combine journalctl with other monitoring tools: ```bash Export to external log management journalctl -o json --since "1 hour ago" | curl -X POST -d @- http://logserver/api Integration with monitoring systems journalctl -p err --since "5 min ago" --no-pager | mail -s "System Errors" admin@company.com ``` Conclusion Mastering journalctl is essential for effective Linux system administration and troubleshooting. This comprehensive guide has covered everything from basic log viewing to advanced filtering techniques, providing you with the knowledge needed to efficiently manage and analyze system logs. Key takeaways from this guide include: - Basic Navigation: Understanding how to view, follow, and navigate through logs using journalctl's built-in pager and options - Effective Filtering: Using time-based, service-specific, and priority-level filters to quickly locate relevant information - Advanced Features: Leveraging JSON output, field-based filtering, and custom formatting for specialized use cases - Maintenance: Implementing proper log rotation, cleanup, and storage management practices - Troubleshooting: Recognizing and resolving common issues with journal access and performance As you continue working with journalctl, remember to: 1. Start with specific filters to reduce noise and improve performance 2. Use appropriate time ranges to focus on relevant events 3. Implement regular maintenance to prevent storage issues 4. Document common queries for your specific environment 5. Monitor system health proactively using automated scripts The systemd journal and journalctl provide powerful capabilities for system monitoring and troubleshooting. By applying the techniques and best practices outlined in this guide, you'll be well-equipped to maintain healthy Linux systems and quickly resolve issues when they arise. Continue practicing with different filtering options and explore advanced features as your needs evolve. The investment in mastering journalctl will pay dividends in improved system reliability and faster problem resolution.