How to show critical chain → systemd-analyze critical-chain
How to Show Critical Chain → systemd-analyze critical-chain
Table of Contents
1. [Introduction](#introduction)
2. [Prerequisites](#prerequisites)
3. [Understanding the Critical Chain](#understanding-the-critical-chain)
4. [Basic Usage of systemd-analyze critical-chain](#basic-usage-of-systemd-analyze-critical-chain)
5. [Advanced Options and Parameters](#advanced-options-and-parameters)
6. [Interpreting Critical Chain Output](#interpreting-critical-chain-output)
7. [Practical Examples and Use Cases](#practical-examples-and-use-cases)
8. [Troubleshooting Common Issues](#troubleshooting-common-issues)
9. [Best Practices and Optimization Tips](#best-practices-and-optimization-tips)
10. [Related systemd-analyze Commands](#related-systemd-analyze-commands)
11. [Conclusion](#conclusion)
Introduction
System boot performance is crucial for both server environments and desktop systems. When your Linux system takes too long to start, identifying the bottlenecks becomes essential for optimization. The `systemd-analyze critical-chain` command is a powerful diagnostic tool that reveals the critical path of service dependencies during system startup, helping you pinpoint exactly which services are causing delays.
This comprehensive guide will teach you how to effectively use `systemd-analyze critical-chain` to analyze boot performance, understand service dependencies, and optimize your system's startup time. Whether you're a system administrator troubleshooting slow servers or a Linux enthusiast wanting to speed up your desktop boot time, this article provides the knowledge and practical examples you need.
By the end of this guide, you'll understand how to read critical chain output, identify performance bottlenecks, and implement effective optimization strategies to reduce boot times significantly.
Prerequisites
Before diving into the critical chain analysis, ensure you have the following:
System Requirements
- Linux distribution with systemd: Most modern Linux distributions (Ubuntu 15.04+, CentOS 7+, Debian 8+, Fedora, openSUSE, Arch Linux)
- systemd version 219 or higher: Check with `systemctl --version`
- Root or sudo access: Required for detailed system analysis
- Basic terminal knowledge: Familiarity with command-line operations
Verification Commands
```bash
Check if systemd is running
systemctl --version
Verify systemd-analyze is available
which systemd-analyze
Check current boot time
systemd-analyze
```
Expected Output
```
Startup finished in 2.841s (kernel) + 8.032s (userspace) = 10.873s
graphical.target reached after 8.032s in userspace
```
Understanding the Critical Chain
What is the Critical Chain?
The critical chain represents the longest path of service dependencies that must complete before the system reaches its target state (usually `graphical.target` or `multi-user.target`). This chain shows the sequence of services that directly impact boot time, as each service in the chain must wait for the previous one to complete.
Key Concepts
Service Dependencies: Services often depend on other services to function correctly. For example, a web server might depend on the network service, which depends on the network interface initialization.
Parallel vs Sequential Loading: While systemd can start many services in parallel, some must wait for others, creating dependency chains.
Target States: The system boots toward specific targets:
- `basic.target`: Minimal system
- `multi-user.target`: Multi-user system without GUI
- `graphical.target`: Full desktop environment
Why Critical Chain Analysis Matters
Understanding the critical chain helps you:
- Identify the actual bottlenecks affecting boot time
- Distinguish between services that run in parallel vs those blocking the boot process
- Focus optimization efforts on services that matter most
- Understand service dependency relationships
Basic Usage of systemd-analyze critical-chain
Simple Critical Chain Command
The most basic usage shows the critical chain for the default target:
```bash
systemd-analyze critical-chain
```
Typical Output Explanation
```
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.
graphical.target @8.032s
└─multi-user.target @8.031s
└─mysql.service @7.542s +488ms
└─network-online.target @7.538s
└─NetworkManager-wait-online.service @2.375s +5.162s
└─NetworkManager.service @2.287s +85ms
└─dbus.service @2.249s
└─basic.target @2.244s
└─sockets.target @2.243s
└─dbus.socket @2.243s
└─sysinit.target @2.241s
└─systemd-timesyncd.service @2.156s +84ms
```
Reading the Output Format
- @X.XXXs: Time when the service became active
- +XXXms: Time the service took to start
- └─: Dependency relationship (tree structure)
- Service names: Individual systemd units in the chain
Advanced Options and Parameters
Targeting Specific Services
Analyze the critical chain for a specific service:
```bash
Show critical chain for a specific service
systemd-analyze critical-chain nginx.service
Show critical chain for a specific target
systemd-analyze critical-chain multi-user.target
```
Time-based Filtering
Filter services that took longer than a specific time:
```bash
Show only services taking more than 100ms
systemd-analyze critical-chain --fuzz=100ms
Show services taking more than 1 second
systemd-analyze critical-chain --fuzz=1s
```
Output Formatting Options
```bash
Show more detailed timing information
systemd-analyze critical-chain --no-pager
Combine with other analyze commands
systemd-analyze time && systemd-analyze critical-chain
```
Analyzing Previous Boots
```bash
List available boots
journalctl --list-boots
Analyze critical chain from a previous boot
systemd-analyze critical-chain --boot=-1
```
Interpreting Critical Chain Output
Understanding Time Values
Absolute Times (@): These show when each service became active relative to the start of userspace initialization. Higher values indicate services that started later in the boot process.
Duration Times (+): These show how long each individual service took to start. Services with high duration values are potential optimization targets.
Identifying Bottlenecks
Look for services with:
1. High duration values (+ times over 1 second)
2. Services near the end of long chains
3. Network-related services (often the biggest bottlenecks)
4. Database or storage services
Example Analysis
```
NetworkManager-wait-online.service @2.375s +5.162s
```
This indicates:
- Service became active at 2.375 seconds
- Service took 5.162 seconds to complete
- This is likely a major bottleneck worth investigating
Common Bottleneck Patterns
Network Services:
```
└─NetworkManager-wait-online.service @2.375s +5.162s
```
Storage Services:
```
└─dev-disk-by\x2duuid-xxxxx.device @15.234s +2.845s
```
Database Services:
```
└─postgresql.service @8.234s +3.456s
```
Practical Examples and Use Cases
Example 1: Desktop System Analysis
For a typical desktop system, you might see:
```bash
systemd-analyze critical-chain
```
```
graphical.target @12.456s
└─display-manager.service @12.234s +221ms
└─multi-user.target @12.233s
└─network-online.target @12.232s
└─NetworkManager-wait-online.service @2.123s +10.109s
└─NetworkManager.service @2.045s +77ms
└─dbus.service @1.987s +57ms
└─basic.target @1.934s
```
Analysis: NetworkManager-wait-online.service is taking over 10 seconds, which is the primary bottleneck.
Solution:
```bash
Disable network wait if not needed
sudo systemctl disable NetworkManager-wait-online.service
Or reduce timeout
sudo systemctl edit NetworkManager-wait-online.service
```
Add to the override file:
```ini
[Service]
TimeoutStartSec=30
```
Example 2: Server System Analysis
Server systems often show different patterns:
```bash
systemd-analyze critical-chain multi-user.target
```
```
multi-user.target @8.234s
└─mysql.service @7.123s +1.111s
└─network-online.target @7.122s
└─systemd-networkd-wait-online.service @1.234s +5.888s
└─systemd-networkd.service @1.123s +110ms
└─systemd-resolved.service @1.012s +110ms
└─basic.target @0.987s
```
Analysis: systemd-networkd-wait-online.service is causing a 5.8-second delay.
Optimization:
```bash
Configure faster network timeout
sudo systemctl edit systemd-networkd-wait-online.service
```
```ini
[Service]
ExecStart=
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --timeout=30
```
Example 3: Identifying Storage Issues
```bash
systemd-analyze critical-chain
```
```
multi-user.target @25.678s
└─local-fs.target @25.677s
└─home.mount @20.123s +5.554s
└─dev-disk-by\x2duuid-12345678.device @18.456s +1.667s
└─systemd-fsck@dev-disk-by\x2duuid-12345678.service @15.234s +3.222s
```
Analysis: File system check (fsck) is taking 3.2 seconds, and mounting /home takes 5.5 seconds.
Investigation Steps:
```bash
Check disk health
sudo smartctl -a /dev/sda
Check file system
sudo fsck -n /dev/disk/by-uuid/12345678
Consider SSD optimization
sudo systemctl enable fstrim.timer
```
Troubleshooting Common Issues
Issue 1: Command Not Found
Problem:
```
bash: systemd-analyze: command not found
```
Solutions:
```bash
Install systemd tools (Ubuntu/Debian)
sudo apt-get install systemd
Install systemd tools (CentOS/RHEL)
sudo yum install systemd
Check PATH
echo $PATH
which systemd-analyze
```
Issue 2: Permission Denied
Problem:
```
Failed to create bus connection: Permission denied
```
Solutions:
```bash
Run with sudo
sudo systemd-analyze critical-chain
Check systemd status
systemctl status systemd-logind
Verify user permissions
groups $USER
```
Issue 3: No Critical Chain Data
Problem:
```
No data available for critical chain analysis
```
Solutions:
```bash
Ensure systemd is the init system
ps -p 1 -o comm=
Check boot logs
journalctl -b 0
Reboot and try again
sudo reboot
```
Issue 4: Incomplete or Truncated Output
Problem: Output appears cut off or incomplete.
Solutions:
```bash
Use --no-pager for full output
systemd-analyze critical-chain --no-pager
Redirect to file for analysis
systemd-analyze critical-chain > boot_analysis.txt
Use less for scrolling
systemd-analyze critical-chain | less
```
Issue 5: Services Not Showing Expected Dependencies
Problem: Critical chain doesn't show expected service relationships.
Investigation:
```bash
Check specific service dependencies
systemctl list-dependencies your-service.service
Analyze service configuration
systemctl cat your-service.service
Check service status
systemctl status your-service.service
```
Best Practices and Optimization Tips
Regular Monitoring
Establish Baseline Measurements:
```bash
Create a boot performance log
echo "$(date): $(systemd-analyze)" >> /var/log/boot-performance.log
Regular critical chain analysis
systemd-analyze critical-chain > /tmp/critical-chain-$(date +%Y%m%d).txt
```
Automated Monitoring Script:
```bash
#!/bin/bash
boot-monitor.sh
LOGFILE="/var/log/boot-analysis.log"
BOOT_TIME=$(systemd-analyze | grep "Startup finished" | awk '{print $4}')
echo "$(date): Boot time: $BOOT_TIME" >> $LOGFILE
if (( $(echo "$BOOT_TIME > 30.0" | bc -l) )); then
echo "WARNING: Slow boot detected ($BOOT_TIME)" >> $LOGFILE
systemd-analyze critical-chain >> $LOGFILE
fi
```
Optimization Strategies
Network Service Optimization:
```bash
Reduce NetworkManager-wait-online timeout
sudo mkdir -p /etc/systemd/system/NetworkManager-wait-online.service.d/
sudo tee /etc/systemd/system/NetworkManager-wait-online.service.d/reduce-timeout.conf <Parallel Service Loading:
```bash
Enable parallel startup for custom services
sudo systemctl edit your-service.service
```
```ini
[Unit]
After=basic.target
Remove unnecessary dependencies to allow parallel startup
```
Storage Optimization:
```bash
Enable SSD optimizations
sudo systemctl enable fstrim.timer
Optimize mount options in /etc/fstab
Add noatime,nodiratime for better performance
/dev/sda1 / ext4 defaults,noatime,nodiratime 0 1
```
Service Management Best Practices
Disable Unnecessary Services:
```bash
List all enabled services
systemctl list-unit-files --type=service --state=enabled
Disable services not needed at boot
sudo systemctl disable bluetooth.service
sudo systemctl disable cups.service
sudo systemctl disable avahi-daemon.service
```
Optimize Service Dependencies:
```bash
Check what requires a slow service
systemctl list-dependencies --reverse slow-service.service
Create service overrides to optimize dependencies
sudo systemctl edit problematic-service.service
```
Performance Monitoring Integration
Integration with System Monitoring:
```bash
Add boot time to system metrics
#!/bin/bash
BOOT_TIME=$(systemd-analyze | awk '/Startup finished/ {print $4}' | sed 's/s//')
echo "boot_time_seconds $BOOT_TIME" > /var/lib/node_exporter/textfile_collector/boot_time.prom
```
Alerting on Slow Boots:
```bash
Systemd service for boot monitoring
sudo tee /etc/systemd/system/boot-monitor.service <Overall Boot Time:
```bash
Basic boot time summary
systemd-analyze
Detailed timing breakdown
systemd-analyze time
```
Service Timing Analysis:
```bash
Show all services with timing
systemd-analyze blame
Show services taking more than 1 second
systemd-analyze blame | grep -E '[0-9]+\.[0-9]+s' | head -10
```
Dependency Visualization:
```bash
Generate SVG boot chart
systemd-analyze plot > boot-chart.svg
Generate dependency graph
systemd-analyze dot | dot -Tsvg > dependencies.svg
```
Security and Configuration Analysis
Security Analysis:
```bash
Analyze service security settings
systemd-analyze security
Check specific service security
systemd-analyze security nginx.service
```
Configuration Verification:
```bash
Verify systemd configuration
systemd-analyze verify /etc/systemd/system/*.service
Check for configuration issues
systemd-analyze cat-config systemd/system.conf
```
Historical Analysis
Boot History:
```bash
List previous boots
journalctl --list-boots
Analyze previous boot
systemd-analyze critical-chain --boot=-1
Compare boot times
for i in {0..4}; do
echo "Boot -$i:"
systemd-analyze --boot=-$i 2>/dev/null || echo "No data"
done
```
Conclusion
The `systemd-analyze critical-chain` command is an invaluable tool for understanding and optimizing Linux boot performance. By following the techniques and strategies outlined in this guide, you can:
- Identify real bottlenecks that impact boot time rather than guessing
- Understand service dependencies and their impact on system startup
- Implement targeted optimizations that provide measurable improvements
- Monitor boot performance over time to catch regressions early
- Troubleshoot boot issues systematically using data-driven analysis
Key Takeaways
1. Focus on the critical chain: Not all slow services impact boot time equally
2. Network services are common bottlenecks: Pay special attention to NetworkManager-wait-online and similar services
3. Regular monitoring is essential: Boot performance can degrade over time
4. Combine multiple analysis tools: Use critical-chain alongside blame, plot, and other systemd-analyze commands
5. Test optimizations carefully: Always verify that changes don't break system functionality
Next Steps
After mastering critical chain analysis, consider exploring:
- Advanced systemd service configuration for custom applications
- Boot optimization for specific use cases (servers, embedded systems, containers)
- Integration with monitoring systems for production environments
- Automated boot performance regression testing in CI/CD pipelines
Remember that boot optimization is an iterative process. Start with the biggest bottlenecks identified by the critical chain analysis, implement changes carefully, and measure the results. With systematic application of these techniques, you can significantly reduce boot times and improve overall system responsiveness.
The investment in understanding and optimizing your system's boot process pays dividends in improved user experience, reduced downtime, and more efficient system administration. Whether you're managing a single desktop or hundreds of servers, the insights provided by `systemd-analyze critical-chain` will help you maintain optimal system performance.