How to analyze boot time → systemd-analyze time

How to Analyze Boot Time → systemd-analyze time Table of Contents 1. [Introduction](#introduction) 2. [Prerequisites](#prerequisites) 3. [Understanding systemd-analyze](#understanding-systemd-analyze) 4. [Basic Boot Time Analysis](#basic-boot-time-analysis) 5. [Advanced Analysis Techniques](#advanced-analysis-techniques) 6. [Interpreting Results](#interpreting-results) 7. [Practical Examples](#practical-examples) 8. [Troubleshooting Common Issues](#troubleshooting-common-issues) 9. [Optimization Strategies](#optimization-strategies) 10. [Best Practices](#best-practices) 11. [Conclusion](#conclusion) Introduction Boot time analysis is crucial for system administrators, developers, and power users who want to optimize their Linux systems for faster startup times. The `systemd-analyze` command is a powerful tool that provides detailed insights into your system's boot process, helping you identify bottlenecks and optimize performance. In this comprehensive guide, you'll learn how to effectively use `systemd-analyze time` and related commands to analyze boot performance, interpret the results, and implement optimization strategies. Whether you're managing servers, desktop systems, or embedded devices, understanding boot time analysis will help you create more responsive and efficient systems. Prerequisites Before diving into boot time analysis, ensure you have the following: System Requirements - A Linux distribution using systemd (most modern distributions) - Root or sudo access for detailed analysis - Basic understanding of Linux command line - Familiarity with systemd concepts (services, units, targets) Checking systemd Availability First, verify that your system uses systemd: ```bash Check if systemd is running systemctl --version Verify systemd is the init system ps -p 1 -o comm= ``` If the output shows "systemd" for the second command, you're ready to proceed. Understanding systemd-analyze The `systemd-analyze` command is part of the systemd suite and provides various analysis tools for system performance. It can examine boot times, service dependencies, security settings, and more. Key Features - Boot time measurement: Analyze overall boot performance - Service timing: Identify slow-starting services - Dependency analysis: Visualize service relationships - Critical path analysis: Find the longest boot sequence - Blame analysis: Identify services causing delays Command Structure ```bash systemd-analyze [OPTIONS...] [COMMAND] [ARGS...] ``` Basic Boot Time Analysis Getting Overall Boot Time The most basic analysis starts with examining the overall boot time: ```bash Display overall boot time breakdown systemd-analyze time ``` Example output: ``` Startup finished in 2.847s (kernel) + 1.983s (initrd) + 8.421s (userspace) = 13.251s graphical.target reached after 8.421s in userspace ``` Understanding the Output The output breaks down boot time into distinct phases: 1. Kernel time: Time spent loading and initializing the kernel 2. initrd time: Time spent in initial RAM disk (if used) 3. Userspace time: Time spent starting userspace services 4. Total time: Complete boot time from power-on to target reached Detailed Timing Information For more detailed information, use the verbose flag: ```bash Detailed boot time analysis systemd-analyze time --no-pager ``` This provides additional context about what each phase encompasses. Advanced Analysis Techniques Service Blame Analysis Identify which services are taking the longest to start: ```bash Show services sorted by startup time systemd-analyze blame ``` Example output: ``` 5.234s NetworkManager.service 2.845s plymouth-quit-wait.service 1.923s systemd-networkd-wait-online.service 1.456s mysql.service 0.987s apache2.service 0.654s ssh.service ``` Critical Chain Analysis Find the critical path that determines overall boot time: ```bash Show the critical chain of services systemd-analyze critical-chain ``` Example output: ``` The time when unit became active or started is printed after the "@" character. The time the unit took to start is printed after the "+" character. graphical.target @8.421s └─multi-user.target @8.420s └─mysql.service @6.964s +1.456s └─network.target @6.963s └─NetworkManager.service @1.729s +5.234s └─dbus.service @1.728s └─basic.target @1.727s ``` Per-Service Analysis Analyze specific services in detail: ```bash Analyze a specific service systemd-analyze critical-chain mysql.service Show service dependencies systemd-analyze dot mysql.service | dot -Tsvg > mysql-deps.svg ``` Boot Chart Generation Create visual representations of the boot process: ```bash Generate boot chart (requires systemd-bootchart) systemd-analyze plot > boot-chart.svg Generate dependency graph systemd-analyze dot | dot -Tsvg > boot-dependencies.svg ``` Interpreting Results Understanding Timing Data When analyzing boot times, consider these factors: Normal Boot Time Ranges - Desktop systems: 15-45 seconds (cold boot) - Servers: 30-120 seconds (depending on services) - Embedded systems: 5-30 seconds - Virtual machines: 10-30 seconds Identifying Problems Look for these warning signs: 1. Individual services > 10 seconds: Usually indicates configuration issues 2. Network services timing out: Often DNS or connectivity problems 3. Database services slow: May indicate corruption or large datasets 4. Graphics services delayed: Driver or hardware issues Reading Critical Chain Output The critical chain shows dependencies and timing: ```bash service-name @start-time +duration ``` - `@start-time`: When the service began starting - `+duration`: How long the service took to start - Indentation shows dependency relationships Analyzing Blame Output The blame command shows absolute startup times: - Focus on services taking > 1-2 seconds - Consider if slow services are necessary for your use case - Check if services can be optimized or delayed Practical Examples Example 1: Desktop System Analysis Let's analyze a typical desktop system: ```bash Initial analysis $ systemd-analyze time Startup finished in 3.2s (kernel) + 2.1s (initrd) + 12.4s (userspace) = 17.7s Identify slow services $ systemd-analyze blame | head -5 8.234s NetworkManager-wait-online.service 3.456s snapd.service 2.123s plymouth-quit-wait.service 1.987s accounts-daemon.service 1.654s udisks2.service Check critical chain $ systemd-analyze critical-chain graphical.target @12.4s └─display-manager.service @12.3s +100ms └─multi-user.target @12.3s └─NetworkManager-wait-online.service @4.0s +8.234s ``` Analysis: The NetworkManager-wait-online service is causing an 8+ second delay, likely waiting for network connectivity that may not be necessary for desktop startup. Example 2: Server System Analysis For a web server setup: ```bash Server boot analysis $ systemd-analyze time Startup finished in 2.8s (kernel) + 45.2s (userspace) = 48.0s Check blame $ systemd-analyze blame | head -5 25.123s mysql.service 8.456s apache2.service 5.234s php7.4-fpm.service 3.987s postfix.service 2.345s fail2ban.service Critical chain analysis $ systemd-analyze critical-chain apache2.service apache2.service @33.579s +8.456s └─mysql.service @8.456s +25.123s └─network-online.target @8.455s ``` Analysis: MySQL is taking 25 seconds to start, which delays Apache. This could indicate database recovery, large datasets, or configuration issues. Example 3: Embedded System Analysis For a Raspberry Pi or similar device: ```bash Embedded system analysis $ systemd-analyze time Startup finished in 4.5s (kernel) + 8.2s (userspace) = 12.7s Quick blame check $ systemd-analyze blame | head -3 3.234s bluetooth.service 2.456s wpa_supplicant.service 1.789s dphys-swapfile.service ``` Analysis: Reasonable boot time for embedded hardware, with wireless and Bluetooth services taking expected time. Troubleshooting Common Issues Slow Network Services Problem: NetworkManager-wait-online or similar services causing delays Solutions: ```bash Disable wait-online if not needed sudo systemctl disable NetworkManager-wait-online.service Or configure timeout sudo systemctl edit NetworkManager-wait-online.service Add: [Service] TimeoutStartSec=30 ``` Database Startup Issues Problem: MySQL, PostgreSQL, or other databases starting slowly Diagnostic steps: ```bash Check database logs sudo journalctl -u mysql.service -n 50 Check database configuration sudo mysql -e "SHOW VARIABLES LIKE 'innodb_buffer_pool_size';" Monitor database startup sudo systemd-analyze critical-chain mysql.service ``` Solutions: - Optimize database configuration - Check for corruption: `sudo mysqlcheck --all-databases` - Consider SSD storage for better I/O performance Graphics and Display Issues Problem: Display manager or graphics services slow to start Diagnostic steps: ```bash Check display manager logs sudo journalctl -u gdm.service -n 30 Check graphics driver status lspci -k | grep -A 2 -E "(VGA|3D)" Analyze graphics service chain systemd-analyze critical-chain display-manager.service ``` Service Timeout Issues Problem: Services timing out during boot Solutions: ```bash Increase timeout for specific service sudo systemctl edit slow-service.service Add: [Service] TimeoutStartSec=300 Check service status systemctl status slow-service.service View detailed logs journalctl -u slow-service.service -f ``` Dependency Loops Problem: Circular dependencies causing boot delays Diagnostic steps: ```bash Check for dependency issues systemd-analyze verify Generate dependency graph systemd-analyze dot > deps.dot dot -Tsvg deps.dot > deps.svg ``` Optimization Strategies Service Management Disable Unnecessary Services ```bash List all enabled services systemctl list-unit-files --state=enabled Disable unnecessary services sudo systemctl disable bluetooth.service sudo systemctl disable cups.service # If no printing needed sudo systemctl disable ModemManager.service # If no mobile broadband ``` Delay Non-Critical Services ```bash Create drop-in directory sudo mkdir -p /etc/systemd/system/non-critical.service.d Create override file sudo tee /etc/systemd/system/non-critical.service.d/override.conf << EOF [Unit] After=graphical.target EOF ``` Parallel Service Startup Enable parallel startup for independent services: ```bash Edit service file sudo systemctl edit custom.service Add: [Service] Type=forking RemainAfterExit=yes ``` Storage Optimization Use SSD Storage - Replace HDD with SSD for dramatic boot time improvements - Enable TRIM for SSD maintenance: ```bash sudo systemctl enable fstrim.timer ``` Optimize File System ```bash Check current mount options mount | grep "on / " Consider noatime option in /etc/fstab for better performance /dev/sda1 / ext4 defaults,noatime 0 1 ``` Kernel Optimization Reduce Kernel Boot Time ```bash Edit GRUB configuration sudo nano /etc/default/grub Add or modify: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash fastboot" Update GRUB sudo update-grub ``` Module Loading Optimization ```bash Check loaded modules lsmod Blacklist unnecessary modules in /etc/modprobe.d/blacklist.conf echo "blacklist pcspkr" | sudo tee -a /etc/modprobe.d/blacklist.conf ``` Network Optimization Configure Network Services ```bash For desktop systems, consider disabling wait-online sudo systemctl disable systemd-networkd-wait-online.service For servers, optimize timeout sudo systemctl edit systemd-networkd-wait-online.service Add: [Service] TimeoutStartSec=30 ``` Best Practices Regular Monitoring Establish Baseline Measurements ```bash Create boot time log echo "$(date): $(systemd-analyze time)" >> /var/log/boot-times.log Weekly analysis script #!/bin/bash systemd-analyze time > /tmp/boot-analysis.txt systemd-analyze blame | head -10 >> /tmp/boot-analysis.txt mail -s "Weekly Boot Analysis" admin@example.com < /tmp/boot-analysis.txt ``` Automated Monitoring ```bash Create systemd service for boot time logging sudo tee /etc/systemd/system/boot-logger.service << EOF [Unit] Description=Boot Time Logger After=graphical.target [Service] Type=oneshot ExecStart=/usr/local/bin/log-boot-time.sh RemainAfterExit=yes [Install] WantedBy=graphical.target EOF ``` Documentation and Change Management Document Optimizations - Keep records of changes made - Note performance improvements - Document service purposes before disabling Version Control Configuration ```bash Track systemd configuration changes cd /etc/systemd sudo git init sudo git add . sudo git commit -m "Initial systemd configuration" ``` Testing and Validation Safe Testing Procedures 1. Backup configurations before making changes 2. Test in virtual machines first 3. Document rollback procedures 4. Monitor system stability after changes Performance Regression Testing ```bash Before optimization systemd-analyze time > before-optimization.txt systemd-analyze blame > before-blame.txt After optimization systemd-analyze time > after-optimization.txt systemd-analyze blame > after-blame.txt Compare results diff before-optimization.txt after-optimization.txt ``` Security Considerations Maintain Security While Optimizing - Don't disable security-related services - Keep system updates current - Monitor for security implications of changes ```bash Check security status systemd-analyze security Verify critical security services are running systemctl status ufw.service systemctl status fail2ban.service ``` Advanced Use Cases Multi-Boot Analysis For systems with multiple boot configurations: ```bash Compare boot times across different kernels Boot with different kernel, then: systemd-analyze time > kernel-5.15-boot.txt Switch kernel and reboot systemd-analyze time > kernel-5.19-boot.txt Compare diff kernel-5.15-boot.txt kernel-5.19-boot.txt ``` Container and Virtualization Analyzing Container Boot Times ```bash For systemd in containers systemd-analyze time --host container-name Docker container analysis docker run --privileged -d systemd-container docker exec container-name systemd-analyze time ``` Virtual Machine Optimization ```bash VM-specific optimizations Disable hardware detection services sudo systemctl disable lvm2-monitor.service Optimize for virtualized environment echo "elevator=noop" | sudo tee -a /etc/default/grub ``` Embedded Systems IoT Device Boot Optimization ```bash Minimal service set for IoT systemctl list-unit-files --state=enabled | grep -v "systemd\|dbus\|network" Disable GUI components sudo systemctl set-default multi-user.target Custom boot target sudo cp /lib/systemd/system/multi-user.target /etc/systemd/system/iot.target ``` Conclusion Boot time analysis using `systemd-analyze time` is an essential skill for Linux system optimization. By understanding how to measure, analyze, and optimize boot performance, you can significantly improve system responsiveness and user experience. Key Takeaways 1. Regular monitoring helps identify performance regressions early 2. Service analysis reveals optimization opportunities 3. Systematic approach to optimization prevents system instability 4. Documentation ensures reproducible improvements 5. Security considerations must be balanced with performance gains Next Steps After mastering boot time analysis, consider exploring: - Advanced systemd features: Timers, sockets, and path units - System profiling tools: perf, iotop, and htop for runtime analysis - Automated deployment: Ansible or similar tools for configuration management - Monitoring solutions: Prometheus, Grafana for ongoing performance tracking Final Recommendations - Start with baseline measurements before making changes - Focus on the biggest bottlenecks first - Test changes in non-production environments - Keep detailed records of modifications - Regularly review and update optimizations By following this comprehensive guide, you now have the knowledge and tools necessary to effectively analyze and optimize boot times on Linux systems using systemd-analyze. Remember that optimization is an iterative process, and regular monitoring will help maintain optimal performance over time. The investment in understanding boot time analysis pays dividends in improved system performance, better user experience, and more efficient resource utilization across your Linux infrastructure.