How to schedule scripts with systemd timers
How to Schedule Scripts with Systemd Timers
Systemd timers have emerged as a powerful and flexible alternative to traditional cron jobs for scheduling scripts and automated tasks on Linux systems. Unlike cron, systemd timers integrate seamlessly with the systemd ecosystem, providing better logging, dependency management, and more sophisticated scheduling options. This comprehensive guide will walk you through everything you need to know about creating, managing, and optimizing systemd timers for your automation needs.
Table of Contents
1. [Introduction to Systemd Timers](#introduction-to-systemd-timers)
2. [Prerequisites and Requirements](#prerequisites-and-requirements)
3. [Understanding Systemd Timer Components](#understanding-systemd-timer-components)
4. [Creating Your First Systemd Timer](#creating-your-first-systemd-timer)
5. [Advanced Timer Configuration](#advanced-timer-configuration)
6. [Practical Examples and Use Cases](#practical-examples-and-use-cases)
7. [Managing and Monitoring Timers](#managing-and-monitoring-timers)
8. [Troubleshooting Common Issues](#troubleshooting-common-issues)
9. [Best Practices and Tips](#best-practices-and-tips)
10. [Conclusion](#conclusion)
Introduction to Systemd Timers
Systemd timers are specialized systemd units that trigger other systemd units (typically services) at specified times or intervals. They offer several advantages over traditional cron jobs:
- Better Integration: Native integration with systemd's logging and monitoring capabilities
- Dependency Management: Support for complex dependencies between services
- Flexible Scheduling: More sophisticated timing options including calendar events and monotonic timers
- Resource Control: Built-in support for resource limits and security restrictions
- Centralized Management: Unified interface for managing all system services and timers
Key Benefits
- Enhanced Logging: All output is automatically captured by journald
- Service Dependencies: Can wait for network, filesystem, or other services
- Security Features: Built-in sandboxing and privilege management
- Monitoring Integration: Native support for system monitoring tools
- Failure Handling: Sophisticated retry and failure handling mechanisms
Prerequisites and Requirements
Before diving into systemd timers, ensure you have:
System Requirements
- A Linux distribution using systemd (most modern distributions)
- Root or sudo access for system-wide timers
- Basic understanding of systemd concepts
- Familiarity with command-line operations
Required Knowledge
- Basic Linux system administration
- Understanding of file permissions and ownership
- Familiarity with text editors (nano, vim, or similar)
- Basic scripting knowledge (bash, python, etc.)
Verification Commands
Check if your system uses systemd:
```bash
Verify systemd is running
systemctl --version
Check systemd status
systemctl status
List existing timers
systemctl list-timers
```
Understanding Systemd Timer Components
Systemd timers consist of two main components that work together:
Timer Unit (.timer file)
The timer unit defines when and how often a task should run. It contains scheduling information and configuration options.
Service Unit (.service file)
The service unit defines what should be executed. It contains the actual command, script, or program to run along with execution parameters.
File Locations
- System-wide timers: `/etc/systemd/system/`
- User timers: `~/.config/systemd/user/`
- System defaults: `/lib/systemd/system/` (don't modify these)
Basic Timer Structure
```ini
[Unit]
Description=Description of what this timer does
Requires=network.target
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
```
Creating Your First Systemd Timer
Let's create a simple timer that runs a backup script daily. This example will demonstrate the fundamental concepts and workflow.
Step 1: Create the Script
First, create a simple backup script:
```bash
sudo mkdir -p /opt/scripts
sudo nano /opt/scripts/backup.sh
```
Add the following content:
```bash
#!/bin/bash
Simple backup script
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/opt/backups"
SOURCE_DIR="/home"
Create backup directory if it doesn't exist
mkdir -p "$BACKUP_DIR"
Create backup
echo "Starting backup at $(date)"
tar -czf "$BACKUP_DIR/backup_$DATE.tar.gz" "$SOURCE_DIR"
echo "Backup completed at $(date)"
Clean up old backups (keep last 7 days)
find "$BACKUP_DIR" -name "backup_*.tar.gz" -mtime +7 -delete
```
Make the script executable:
```bash
sudo chmod +x /opt/scripts/backup.sh
```
Step 2: Create the Service Unit
Create the service file:
```bash
sudo nano /etc/systemd/system/backup.service
```
Add the following configuration:
```ini
[Unit]
Description=Daily Backup Service
After=network.target
[Service]
Type=oneshot
User=root
ExecStart=/opt/scripts/backup.sh
StandardOutput=journal
StandardError=journal
```
Step 3: Create the Timer Unit
Create the timer file:
```bash
sudo nano /etc/systemd/system/backup.timer
```
Add the timer configuration:
```ini
[Unit]
Description=Run backup script daily
Requires=backup.service
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
```
Step 4: Enable and Start the Timer
Reload systemd and enable the timer:
```bash
Reload systemd configuration
sudo systemctl daemon-reload
Enable the timer to start at boot
sudo systemctl enable backup.timer
Start the timer immediately
sudo systemctl start backup.timer
Check timer status
sudo systemctl status backup.timer
```
Step 5: Verify the Timer
Check if the timer is active and scheduled:
```bash
List all active timers
systemctl list-timers
Check specific timer details
systemctl show backup.timer
View timer logs
journalctl -u backup.timer
```
Advanced Timer Configuration
Systemd timers offer sophisticated scheduling options beyond basic intervals. Let's explore advanced configuration techniques.
Calendar Event Expressions
Systemd supports flexible calendar expressions for complex scheduling:
```ini
Every day at 2:30 AM
OnCalendar=--* 02:30:00
Every Monday at 9:00 AM
OnCalendar=Mon --* 09:00:00
Every 15 minutes
OnCalendar=*:0/15
First day of every month at midnight
OnCalendar=--01 00:00:00
Weekdays at 8:30 AM
OnCalendar=Mon..Fri --* 08:30:00
Multiple times per day
OnCalendar=--* 06:00,12:00,18:00
```
Monotonic Timers
For interval-based scheduling relative to system events:
```ini
15 minutes after boot
OnBootSec=15min
30 seconds after the timer is activated
OnStartupSec=30sec
1 hour after the service last finished
OnUnitActiveSec=1h
5 minutes after the service becomes inactive
OnUnitInactiveSec=5min
```
Advanced Timer Example
Here's a sophisticated timer configuration for a system maintenance script:
```ini
[Unit]
Description=System Maintenance Timer
Documentation=man:systemd.timer(5)
[Timer]
Run every Sunday at 3:00 AM
OnCalendar=Sun --* 03:00:00
If system was off, run within 1 hour of next boot
Persistent=true
Add random delay up to 30 minutes to avoid system load spikes
RandomizedDelaySec=30min
Ensure timer accuracy within 1 minute
AccuracySec=1min
[Install]
WantedBy=timers.target
```
Service Configuration Options
Enhance your service units with advanced options:
```ini
[Unit]
Description=Advanced Backup Service
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User=backup
Group=backup
ExecStart=/opt/scripts/advanced-backup.sh
ExecStartPre=/bin/mkdir -p /var/log/backup
ExecStartPost=/opt/scripts/backup-notification.sh
Environment variables
Environment=BACKUP_RETENTION=30
Environment=BACKUP_COMPRESSION=gzip
Working directory
WorkingDirectory=/opt/backup
Timeout settings
TimeoutStartSec=3600
TimeoutStopSec=60
Resource limits
MemoryMax=1G
CPUQuota=50%
Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/opt/backups /var/log/backup
Restart policy for long-running tasks
Restart=on-failure
RestartSec=300
```
Practical Examples and Use Cases
Let's explore several real-world scenarios where systemd timers excel.
Example 1: Log Rotation and Cleanup
Create a timer for custom log rotation:
Service file (`/etc/systemd/system/log-cleanup.service`):
```ini
[Unit]
Description=Custom Log Cleanup Service
After=local-fs.target
[Service]
Type=oneshot
User=root
ExecStart=/bin/bash -c 'find /var/log/myapp -name "*.log" -mtime +30 -delete'
ExecStart=/bin/bash -c 'find /var/log/myapp -name "*.log.gz" -mtime +90 -delete'
StandardOutput=journal
StandardError=journal
```
Timer file (`/etc/systemd/system/log-cleanup.timer`):
```ini
[Unit]
Description=Daily log cleanup
Requires=log-cleanup.service
[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=3600
[Install]
WantedBy=timers.target
```
Example 2: Database Backup with Notifications
Backup script (`/opt/scripts/db-backup.sh`):
```bash
#!/bin/bash
DB_NAME="myapp"
DB_USER="backup_user"
BACKUP_DIR="/opt/db-backups"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/${DB_NAME}_$DATE.sql.gz"
Create backup directory
mkdir -p "$BACKUP_DIR"
Perform backup
if pg_dump -U "$DB_USER" "$DB_NAME" | gzip > "$BACKUP_FILE"; then
echo "Database backup successful: $BACKUP_FILE"
# Send success notification
curl -X POST "https://hooks.slack.com/your-webhook-url" \
-H 'Content-type: application/json' \
--data "{\"text\":\"Database backup completed successfully: $BACKUP_FILE\"}"
# Clean up old backups (keep 14 days)
find "$BACKUP_DIR" -name "${DB_NAME}_*.sql.gz" -mtime +14 -delete
exit 0
else
echo "Database backup failed!"
# Send failure notification
curl -X POST "https://hooks.slack.com/your-webhook-url" \
-H 'Content-type: application/json' \
--data "{\"text\":\"❌ Database backup FAILED for $DB_NAME\"}"
exit 1
fi
```
Service file (`/etc/systemd/system/db-backup.service`):
```ini
[Unit]
Description=Database Backup Service
After=postgresql.service
Requires=postgresql.service network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User=postgres
Group=postgres
ExecStart=/opt/scripts/db-backup.sh
StandardOutput=journal
StandardError=journal
Security enhancements
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/opt/db-backups
Timeout for large databases
TimeoutStartSec=7200
```
Timer file (`/etc/systemd/system/db-backup.timer`):
```ini
[Unit]
Description=Database backup timer
Requires=db-backup.service
[Timer]
Run at 2:00 AM daily
OnCalendar=--* 02:00:00
Persistent=true
Add randomization to avoid conflicts
RandomizedDelaySec=600
[Install]
WantedBy=timers.target
```
Example 3: System Health Monitoring
Monitoring script (`/opt/scripts/health-check.py`):
```python
#!/usr/bin/env python3
import psutil
import json
import subprocess
import sys
from datetime import datetime
def check_system_health():
health_data = {
'timestamp': datetime.now().isoformat(),
'cpu_percent': psutil.cpu_percent(interval=1),
'memory_percent': psutil.virtual_memory().percent,
'disk_usage': {},
'load_average': psutil.getloadavg(),
'alerts': []
}
# Check disk usage for all mounted filesystems
for partition in psutil.disk_partitions():
try:
usage = psutil.disk_usage(partition.mountpoint)
percent_used = (usage.used / usage.total) * 100
health_data['disk_usage'][partition.mountpoint] = percent_used
if percent_used > 90:
health_data['alerts'].append(f"Disk {partition.mountpoint} is {percent_used:.1f}% full")
except PermissionError:
continue
# Check memory usage
if health_data['memory_percent'] > 90:
health_data['alerts'].append(f"Memory usage is {health_data['memory_percent']:.1f}%")
# Check CPU usage
if health_data['cpu_percent'] > 80:
health_data['alerts'].append(f"CPU usage is {health_data['cpu_percent']:.1f}%")
# Log results
print(json.dumps(health_data, indent=2))
# Send alerts if any critical issues
if health_data['alerts']:
print("CRITICAL ALERTS DETECTED:", file=sys.stderr)
for alert in health_data['alerts']:
print(f" - {alert}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
sys.exit(check_system_health())
```
Service file (`/etc/systemd/system/health-check.service`):
```ini
[Unit]
Description=System Health Check
After=multi-user.target
[Service]
Type=oneshot
User=monitoring
Group=monitoring
ExecStart=/usr/bin/python3 /opt/scripts/health-check.py
StandardOutput=journal
StandardError=journal
Install required packages in ExecStartPre if needed
ExecStartPre=/bin/bash -c 'pip3 show psutil >/dev/null || pip3 install psutil'
```
Timer file (`/etc/systemd/system/health-check.timer`):
```ini
[Unit]
Description=System health monitoring
Requires=health-check.service
[Timer]
Run every 15 minutes
OnCalendar=*:0/15
Persistent=true
[Install]
WantedBy=timers.target
```
Managing and Monitoring Timers
Effective management and monitoring of systemd timers is crucial for maintaining reliable automation.
Essential Management Commands
```bash
List all timers (active and inactive)
systemctl list-timers --all
Show detailed timer information
systemctl show backup.timer
Check timer status
systemctl status backup.timer
Start/stop/restart timers
systemctl start backup.timer
systemctl stop backup.timer
systemctl restart backup.timer
Enable/disable timers
systemctl enable backup.timer
systemctl disable backup.timer
Reload configuration after changes
systemctl daemon-reload
```
Monitoring Timer Execution
```bash
View timer logs
journalctl -u backup.timer
View service logs
journalctl -u backup.service
Follow logs in real-time
journalctl -u backup.service -f
View logs for specific time period
journalctl -u backup.service --since "2024-01-01" --until "2024-01-02"
View logs with specific priority
journalctl -u backup.service -p err
```
Timer Analysis and Debugging
```bash
Check when timer will run next
systemctl list-timers backup.timer
Analyze timer configuration
systemd-analyze calendar "Mon --* 09:00:00"
Test service execution manually
systemctl start backup.service
Check service dependencies
systemctl list-dependencies backup.service
```
Creating Monitoring Dashboards
For production environments, consider integrating timer monitoring with your existing monitoring stack:
Prometheus metrics collection script:
```bash
#!/bin/bash
/opt/scripts/timer-metrics.sh
METRICS_FILE="/var/lib/prometheus/node-exporter/systemd-timers.prom"
{
echo "# HELP systemd_timer_last_trigger_seconds Seconds since last trigger"
echo "# TYPE systemd_timer_last_trigger_seconds gauge"
systemctl list-timers --all --no-pager --plain | tail -n +2 | while read -r line; do
if [[ -n "$line" ]]; then
timer_name=$(echo "$line" | awk '{print $NF}')
last_trigger=$(systemctl show "$timer_name" --property=LastTriggerUSec --value)
if [[ "$last_trigger" != "0" ]]; then
last_trigger_sec=$((last_trigger / 1000000))
current_sec=$(date +%s)
seconds_since=$((current_sec - last_trigger_sec))
echo "systemd_timer_last_trigger_seconds{timer=\"$timer_name\"} $seconds_since"
fi
fi
done
} > "$METRICS_FILE.tmp" && mv "$METRICS_FILE.tmp" "$METRICS_FILE"
```
Troubleshooting Common Issues
Understanding common problems and their solutions will help you maintain robust timer-based automation.
Timer Not Running
Symptoms: Timer appears enabled but never executes
Common Causes and Solutions:
1. Timer not started:
```bash
systemctl start your-timer.timer
```
2. Service file issues:
```bash
# Check service syntax
systemd-analyze verify /etc/systemd/system/your-service.service
# Test service manually
systemctl start your-service.service
systemctl status your-service.service
```
3. Calendar expression errors:
```bash
# Test calendar expressions
systemd-analyze calendar "your-calendar-expression"
```
Permission Issues
Symptoms: Service fails with permission denied errors
Solutions:
```bash
Check file permissions
ls -la /path/to/your/script
Fix script permissions
chmod +x /path/to/your/script
Check service user context
systemctl show your-service.service | grep User
Run service with appropriate user
sudo -u service-user /path/to/your/script
```
Service Timeout Issues
Symptoms: Service killed due to timeout
Solutions:
```ini
[Service]
Increase timeout (default is usually 90 seconds)
TimeoutStartSec=1800
TimeoutStopSec=300
For long-running tasks, consider Type=forking
Type=forking
```
Dependency Problems
Symptoms: Service fails because required resources aren't available
Solutions:
```ini
[Unit]
Wait for network
After=network-online.target
Wants=network-online.target
Wait for filesystem
After=local-fs.target
Requires=local-fs.target
Wait for specific services
After=postgresql.service
Requires=postgresql.service
```
Logging and Output Issues
Symptoms: Can't find service output or logs
Solutions:
```ini
[Service]
Ensure output goes to journal
StandardOutput=journal
StandardError=journal
Or redirect to specific files
StandardOutput=file:/var/log/myservice.log
StandardError=file:/var/log/myservice-error.log
```
Common Debugging Commands
```bash
Check systemd journal for errors
journalctl -xe
Verify unit file syntax
systemd-analyze verify /etc/systemd/system/your-timer.timer
Check timer calendar calculations
systemd-analyze calendar "Mon --* 09:00:00"
List failed services
systemctl --failed
Check service environment
systemctl show-environment
Debug service execution
systemd-run --uid=your-user --gid=your-group /path/to/your/script
```
Best Practices and Tips
Following these best practices will help you create robust, maintainable, and secure timer-based automation.
Security Best Practices
1. Use Dedicated Users:
```bash
# Create dedicated service user
sudo useradd -r -s /bin/false backup-user
sudo usermod -a -G backup backup-user
```
2. Implement Principle of Least Privilege:
```ini
[Service]
User=backup-user
Group=backup-group
# Restrict filesystem access
ProtectSystem=strict
ReadWritePaths=/opt/backups
# Prevent privilege escalation
NoNewPrivileges=true
# Use private tmp
PrivateTmp=true
```
3. Secure Script Permissions:
```bash
# Set restrictive permissions
chmod 750 /opt/scripts/backup.sh
chown root:backup-group /opt/scripts/backup.sh
```
Performance Optimization
1. Use RandomizedDelaySec to avoid system load spikes:
```ini
[Timer]
OnCalendar=daily
RandomizedDelaySec=3600
```
2. Set Resource Limits:
```ini
[Service]
MemoryMax=512M
CPUQuota=25%
IOWeight=100
```
3. Optimize I/O Operations:
```bash
# Use ionice for I/O intensive tasks
ExecStart=/usr/bin/ionice -c 3 /opt/scripts/backup.sh
```
Reliability and Monitoring
1. Implement Health Checks:
```ini
[Service]
ExecStart=/opt/scripts/backup.sh
ExecStartPost=/opt/scripts/verify-backup.sh
```
2. Set Up Notifications:
```bash
#!/bin/bash
# In your script
if ! backup_command; then
systemd-cat -t backup-script -p err echo "Backup failed"
# Send notification
mail -s "Backup Failed" admin@example.com < /dev/null
exit 1
fi
```
3. Use Persistent Timers for critical tasks:
```ini
[Timer]
OnCalendar=daily
Persistent=true
```
Maintenance and Documentation
1. Document Your Timers:
```ini
[Unit]
Description=Daily database backup for production system
Documentation=https://wiki.company.com/backups
```
2. Version Control Configuration:
```bash
# Keep timer configs in version control
git add /etc/systemd/system/backup.*
git commit -m "Add daily backup timer"
```
3. Regular Testing:
```bash
# Test timers regularly
systemctl start backup.service
journalctl -u backup.service --since "1 hour ago"
```
Migration from Cron
When migrating from cron to systemd timers:
1. Cron to Calendar Expression Conversion:
```bash
# Cron: 0 2 *
# Systemd: --* 02:00:00
# Cron: /15 *
# Systemd: *:0/15
# Cron: 0 9 1-5
# Systemd: Mon..Fri --* 09:00:00
```
2. Environment Variables:
```ini
[Service]
# Cron sets minimal environment
# Explicitly set required variables
Environment=PATH=/usr/local/bin:/usr/bin:/bin
Environment=HOME=/home/service-user
```
3. Working Directory:
```ini
[Service]
# Cron runs from user's home directory
WorkingDirectory=/home/service-user
```
Conclusion
Systemd timers represent a powerful evolution in Linux task scheduling, offering significant advantages over traditional cron jobs. Through this comprehensive guide, you've learned how to:
- Create and configure basic systemd timers and services
- Implement advanced scheduling patterns and calendar expressions
- Apply security best practices and resource management
- Monitor and troubleshoot timer-based automation
- Migrate existing cron jobs to systemd timers
Key Takeaways
1. Integration Benefits: Systemd timers provide superior integration with modern Linux systems, offering better logging, dependency management, and security features.
2. Flexibility: Calendar expressions and monotonic timers offer more scheduling flexibility than traditional cron syntax.
3. Security: Built-in sandboxing and privilege management features enhance system security.
4. Monitoring: Native integration with journald and systemd monitoring tools simplifies troubleshooting and maintenance.
Next Steps
To further enhance your systemd timer implementation:
1. Explore Advanced Features: Investigate systemd's path units, socket activation, and complex dependency chains.
2. Implement Monitoring: Set up comprehensive monitoring and alerting for your critical timers.
3. Automate Deployment: Use configuration management tools like Ansible or Puppet to deploy and manage timers across multiple systems.
4. Performance Tuning: Monitor resource usage and optimize timer configurations for your specific workloads.
5. Community Resources: Engage with the systemd community and contribute to best practices documentation.
By mastering systemd timers, you'll have a robust foundation for implementing reliable, secure, and maintainable automation in modern Linux environments. The investment in learning these concepts will pay dividends in system reliability and administrative efficiency.
Remember that good automation is not just about scheduling tasks—it's about creating maintainable, monitorable, and reliable systems that enhance rather than complicate your infrastructure management.