How to track file changes with inotifywait
How to Track File Changes with inotifywait
File system monitoring is a critical aspect of system administration, security auditing, and automated workflow management. The `inotifywait` utility provides a powerful and efficient way to monitor file system events in real-time on Linux systems. This comprehensive guide will walk you through everything you need to know about using `inotifywait` to track file changes, from basic monitoring to advanced scripting techniques.
Table of Contents
1. [Introduction to inotifywait](#introduction-to-inotifywait)
2. [Prerequisites and Installation](#prerequisites-and-installation)
3. [Basic Usage and Syntax](#basic-usage-and-syntax)
4. [Understanding Event Types](#understanding-event-types)
5. [Practical Examples](#practical-examples)
6. [Advanced Monitoring Techniques](#advanced-monitoring-techniques)
7. [Creating Monitoring Scripts](#creating-monitoring-scripts)
8. [Performance Considerations](#performance-considerations)
9. [Troubleshooting Common Issues](#troubleshooting-common-issues)
10. [Best Practices](#best-practices)
11. [Conclusion](#conclusion)
Introduction to inotifywait
The `inotifywait` command is part of the inotify-tools package and leverages the Linux kernel's inotify subsystem to monitor file system events efficiently. Unlike traditional polling methods that repeatedly check file status, `inotifywait` uses kernel-level notifications, making it extremely efficient for real-time file system monitoring.
This tool is invaluable for various scenarios including:
- Automated backup systems that trigger when files change
- Security monitoring to detect unauthorized file modifications
- Development workflows that rebuild projects when source files change
- Log file monitoring and analysis
- Configuration file change detection
Prerequisites and Installation
System Requirements
Before using `inotifywait`, ensure your system meets these requirements:
- Linux operating system with kernel version 2.6.13 or later
- inotify support enabled in the kernel (enabled by default in most distributions)
- Sufficient user permissions to read the files and directories you want to monitor
Installing inotify-tools
The installation process varies depending on your Linux distribution:
Ubuntu/Debian:
```bash
sudo apt-get update
sudo apt-get install inotify-tools
```
CentOS/RHEL/Fedora:
```bash
For CentOS/RHEL 7 and earlier
sudo yum install inotify-tools
For CentOS/RHEL 8+ and Fedora
sudo dnf install inotify-tools
```
Arch Linux:
```bash
sudo pacman -S inotify-tools
```
From Source:
```bash
wget http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz
tar -xzf inotify-tools-3.14.tar.gz
cd inotify-tools-3.14
./configure
make
sudo make install
```
Verifying Installation
After installation, verify that `inotifywait` is available:
```bash
inotifywait --help
```
You should see the help output displaying available options and usage information.
Basic Usage and Syntax
The basic syntax for `inotifywait` is:
```bash
inotifywait [options] file1 [file2] [file3] [...]
```
Essential Command Options
Here are the most commonly used options:
| Option | Description |
|--------|-------------|
| `-m, --monitor` | Keep listening for events (don't exit after first event) |
| `-r, --recursive` | Watch directories recursively |
| `-e, --event` | Specify which events to listen for |
| `--format` | Specify output format |
| `-q, --quiet` | Print less (only print events) |
| `--timefmt` | Specify time format for %T in --format |
| `--exclude` | Exclude files matching the specified pattern |
| `--include` | Include files matching the specified pattern |
Simple Example
Let's start with a basic example that monitors a single file:
```bash
inotifywait -m /path/to/your/file.txt
```
This command will continuously monitor `file.txt` and report any events that occur on it.
Understanding Event Types
`inotifywait` can monitor various types of file system events. Understanding these events is crucial for effective monitoring:
Common Event Types
| Event | Description |
|-------|-------------|
| `access` | File was accessed (read) |
| `modify` | File was modified |
| `attrib` | Metadata changed (permissions, timestamps, etc.) |
| `close_write` | File opened for writing was closed |
| `close_nowrite` | File not opened for writing was closed |
| `close` | File was closed (regardless of read/write mode) |
| `open` | File was opened |
| `moved_to` | File moved to watched directory |
| `moved_from` | File moved from watched directory |
| `move` | File was moved |
| `create` | File/directory created in watched directory |
| `delete` | File/directory deleted from watched directory |
| `delete_self` | Watched file/directory was deleted |
| `unmount` | File system containing watched file was unmounted |
Monitoring Specific Events
You can specify which events to monitor using the `-e` option:
```bash
Monitor only file modifications
inotifywait -m -e modify /path/to/directory
Monitor multiple event types
inotifywait -m -e create,delete,modify /path/to/directory
Monitor all events (default behavior)
inotifywait -m -e all /path/to/directory
```
Practical Examples
Example 1: Basic File Monitoring
Monitor changes to a configuration file:
```bash
inotifywait -m -e modify /etc/nginx/nginx.conf
```
This will output something like:
```
/etc/nginx/nginx.conf MODIFY
```
Example 2: Directory Monitoring with Recursive Option
Monitor all changes in a directory and its subdirectories:
```bash
inotifywait -m -r -e create,delete,modify /home/user/documents/
```
Example 3: Formatted Output
Use custom formatting to get more detailed information:
```bash
inotifywait -m -r --format '%T %w%f %e' --timefmt '%Y-%m-%d %H:%M:%S' /home/user/projects/
```
This produces output like:
```
2024-01-15 14:30:25 /home/user/projects/file.txt MODIFY
2024-01-15 14:30:26 /home/user/projects/newfile.txt CREATE
```
Example 4: Excluding Specific Files
Monitor a directory while excluding temporary files:
```bash
inotifywait -m -r --exclude '\.tmp$|\.swp$' /home/user/documents/
```
Example 5: Including Only Specific File Types
Monitor only Python files in a project directory:
```bash
inotifywait -m -r --include '\.py$' -e modify /home/user/python-project/
```
Advanced Monitoring Techniques
Monitoring Multiple Directories
You can monitor multiple directories simultaneously:
```bash
inotifywait -m -r /var/log/ /etc/ /home/user/important/
```
Using Event Masks
For more precise control, you can use event masks:
```bash
Monitor for close events on files opened for writing
inotifywait -m -e close_write /var/log/application.log
```
Timeout Functionality
Set a timeout to stop monitoring after a specified period:
```bash
Monitor for 60 seconds then exit
timeout 60 inotifywait -m -e modify /path/to/file
```
Background Monitoring
Run `inotifywait` in the background and redirect output to a log file:
```bash
inotifywait -m -r -e create,delete,modify --format '%T %w%f %e' \
--timefmt '%Y-%m-%d %H:%M:%S' /important/directory/ > /var/log/file-changes.log 2>&1 &
```
Creating Monitoring Scripts
Basic Monitoring Script
Here's a simple script that monitors a directory and performs actions based on events:
```bash
#!/bin/bash
WATCH_DIR="/home/user/watched_folder"
LOG_FILE="/var/log/file_monitor.log"
Function to log messages
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}
Start monitoring
log_message "Starting file monitoring on $WATCH_DIR"
inotifywait -m -r -e create,delete,modify --format '%w%f %e' "$WATCH_DIR" | while read FILE EVENT
do
case $EVENT in
CREATE)
log_message "File created: $FILE"
# Add your custom actions here
;;
DELETE)
log_message "File deleted: $FILE"
# Add your custom actions here
;;
MODIFY)
log_message "File modified: $FILE"
# Add your custom actions here
;;
esac
done
```
Advanced Backup Script
This script automatically backs up files when they're modified:
```bash
#!/bin/bash
SOURCE_DIR="/home/user/documents"
BACKUP_DIR="/backup/documents"
LOG_FILE="/var/log/auto_backup.log"
Ensure backup directory exists
mkdir -p "$BACKUP_DIR"
Function to create backup
backup_file() {
local file="$1"
local relative_path="${file#$SOURCE_DIR/}"
local backup_path="$BACKUP_DIR/$relative_path"
local backup_dir=$(dirname "$backup_path")
# Create backup directory if it doesn't exist
mkdir -p "$backup_dir"
# Copy file with timestamp
local timestamp=$(date '+%Y%m%d_%H%M%S')
local backup_file="${backup_path}.${timestamp}"
if cp "$file" "$backup_file"; then
echo "$(date '+%Y-%m-%d %H:%M:%S') - Backed up: $file -> $backup_file" >> "$LOG_FILE"
else
echo "$(date '+%Y-%m-%d %H:%M:%S') - Backup failed: $file" >> "$LOG_FILE"
fi
}
Start monitoring
echo "$(date '+%Y-%m-%d %H:%M:%S') - Starting automatic backup monitoring" >> "$LOG_FILE"
inotifywait -m -r -e close_write --format '%w%f' "$SOURCE_DIR" | while read FILE
do
# Only backup regular files (not directories)
if [[ -f "$FILE" ]]; then
backup_file "$FILE"
fi
done
```
Security Monitoring Script
This script monitors sensitive directories and sends alerts:
```bash
#!/bin/bash
WATCH_DIRS=("/etc" "/var/log" "/home/admin")
ALERT_EMAIL="admin@example.com"
LOG_FILE="/var/log/security_monitor.log"
Function to send alert
send_alert() {
local message="$1"
echo "$message" | mail -s "Security Alert: File System Change" "$ALERT_EMAIL"
echo "$(date '+%Y-%m-%d %H:%M:%S') - ALERT: $message" >> "$LOG_FILE"
}
Function to check if file is sensitive
is_sensitive_file() {
local file="$1"
# Define patterns for sensitive files
if [[ "$file" =~ /etc/passwd|/etc/shadow|/etc/sudoers|\.ssh/ ]]; then
return 0
fi
return 1
}
Start monitoring all directories
for dir in "${WATCH_DIRS[@]}"; do
if [[ -d "$dir" ]]; then
(
inotifywait -m -r -e create,delete,modify,attrib \
--format '%w%f %e' "$dir" | while read FILE EVENT
do
if is_sensitive_file "$FILE"; then
send_alert "Sensitive file $FILE was $EVENT"
else
echo "$(date '+%Y-%m-%d %H:%M:%S') - $FILE $EVENT" >> "$LOG_FILE"
fi
done
) &
fi
done
Wait for all background processes
wait
```
Performance Considerations
Resource Usage
`inotifywait` is generally lightweight, but consider these factors:
1. Number of Watches: Each watched file or directory consumes a small amount of kernel memory
2. Recursive Monitoring: Watching large directory trees can consume significant resources
3. Event Frequency: High-frequency events can impact performance
Optimizing Performance
Limit Watch Scope
Instead of monitoring entire file systems, be specific:
```bash
Good: Monitor specific directory
inotifywait -m -r /var/log/application/
Less optimal: Monitor entire /var
inotifywait -m -r /var/
```
Use Appropriate Events
Only monitor events you actually need:
```bash
Monitor only file modifications, not access
inotifywait -m -e modify /path/to/file
```
Exclude Unnecessary Files
Use exclusion patterns to reduce noise:
```bash
inotifywait -m -r --exclude '\.(tmp|swp|log)$' /project/directory/
```
System Limits
Check and adjust system limits if needed:
```bash
Check current limits
cat /proc/sys/fs/inotify/max_user_watches
cat /proc/sys/fs/inotify/max_user_instances
Increase limits (temporary)
echo 524288 | sudo tee /proc/sys/fs/inotify/max_user_watches
Make permanent by adding to /etc/sysctl.conf
echo 'fs.inotify.max_user_watches=524288' | sudo tee -a /etc/sysctl.conf
```
Troubleshooting Common Issues
Issue 1: "No space left on device" Error
Problem: This error occurs when you've exceeded the maximum number of inotify watches.
Solution:
```bash
Check current usage
find /proc/*/fd -lname anon_inode:inotify -print 2>/dev/null | wc -l
Increase the limit
echo 'fs.inotify.max_user_watches=524288' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
```
Issue 2: Events Not Being Detected
Problem: `inotifywait` doesn't detect expected file changes.
Possible Causes and Solutions:
1. File system doesn't support inotify:
```bash
# Check file system type
df -T /path/to/watch
```
Network file systems (NFS, CIFS) may not support inotify.
2. Insufficient permissions:
```bash
# Ensure read permissions on watched files/directories
ls -la /path/to/watch
```
3. File moved instead of modified:
Some applications save files by creating new files and moving them, which generates `moved_to` events instead of `modify`.
Issue 3: Too Many Events
Problem: `inotifywait` generates excessive output.
Solution:
```bash
Use more specific event filtering
inotifywait -m -e close_write /path/to/watch
Add exclusion patterns
inotifywait -m --exclude '\.tmp$' /path/to/watch
```
Issue 4: Script Stops Unexpectedly
Problem: Monitoring script terminates without obvious reason.
Solutions:
1. Add error handling:
```bash
#!/bin/bash
set -e # Exit on error
trap 'echo "Script interrupted"; exit 1' INT TERM
while true; do
inotifywait -m /path/to/watch || {
echo "inotifywait failed, restarting..."
sleep 5
}
done
```
2. Use systemd for process management:
```ini
# /etc/systemd/system/file-monitor.service
[Unit]
Description=File Monitor Service
After=network.target
[Service]
Type=simple
User=monitor
ExecStart=/usr/local/bin/monitor-script.sh
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
```
Issue 5: High CPU Usage
Problem: `inotifywait` or associated scripts consume too much CPU.
Solutions:
1. Reduce monitoring scope
2. Optimize event processing
3. Add delays in processing loops:
```bash
inotifywait -m /path | while read event; do
process_event "$event"
sleep 0.1 # Small delay to prevent CPU spinning
done
```
Best Practices
1. Use Specific Event Types
Only monitor events you actually need:
```bash
Instead of monitoring all events
inotifywait -m -e all /path/to/watch
Be specific
inotifywait -m -e close_write,create,delete /path/to/watch
```
2. Implement Proper Logging
Always log monitoring activities:
```bash
#!/bin/bash
LOG_FILE="/var/log/file-monitor.log"
exec 1> >(tee -a "$LOG_FILE")
exec 2>&1
echo "$(date): Starting file monitoring"
inotifywait -m /path/to/watch
```
3. Handle Signals Gracefully
Implement proper signal handling:
```bash
#!/bin/bash
cleanup() {
echo "Shutting down file monitoring..."
kill $INOTIFY_PID 2>/dev/null
exit 0
}
trap cleanup INT TERM
inotifywait -m /path/to/watch &
INOTIFY_PID=$!
wait $INOTIFY_PID
```
4. Use Configuration Files
For complex monitoring setups, use configuration files:
```bash
#!/bin/bash
CONFIG_FILE="/etc/file-monitor.conf"
Read configuration
source "$CONFIG_FILE"
for dir in "${WATCH_DIRS[@]}"; do
inotifywait -m -r -e "${EVENTS}" "$dir" &
done
wait
```
5. Implement Rate Limiting
Prevent overwhelming the system with too many events:
```bash
#!/bin/bash
LAST_EVENT_TIME=0
MIN_INTERVAL=1 # Minimum seconds between processing
inotifywait -m /path | while read event; do
CURRENT_TIME=$(date +%s)
if (( CURRENT_TIME - LAST_EVENT_TIME >= MIN_INTERVAL )); then
process_event "$event"
LAST_EVENT_TIME=$CURRENT_TIME
fi
done
```
6. Monitor Script Health
Implement health checks for long-running monitoring scripts:
```bash
#!/bin/bash
HEALTH_FILE="/tmp/monitor-health"
HEALTH_INTERVAL=300 # 5 minutes
Update health file periodically
(
while true; do
echo "$(date): Monitor healthy" > "$HEALTH_FILE"
sleep $HEALTH_INTERVAL
done
) &
Main monitoring loop
inotifywait -m /path/to/watch
```
7. Use Appropriate File Permissions
Ensure monitoring scripts run with minimal necessary privileges:
```bash
Create dedicated user for monitoring
sudo useradd -r -s /bin/false monitor
Set appropriate file ownership
sudo chown monitor:monitor /usr/local/bin/monitor-script.sh
sudo chmod 750 /usr/local/bin/monitor-script.sh
```
8. Document Your Monitoring Setup
Maintain clear documentation:
```bash
#!/bin/bash
File Monitor Script
Purpose: Monitor /var/log for security events
Author: System Administrator
Last Modified: 2024-01-15
Events monitored: create, delete, modify
Notifications: Email alerts for critical files
Logs: /var/log/security-monitor.log
Usage: ./monitor-script.sh
Service: systemctl start file-monitor
```
Conclusion
The `inotifywait` utility is a powerful and efficient tool for real-time file system monitoring on Linux systems. Throughout this comprehensive guide, we've covered everything from basic usage to advanced scripting techniques, providing you with the knowledge and tools needed to implement robust file monitoring solutions.
Key Takeaways
1. Efficiency: `inotifywait` uses kernel-level notifications, making it far more efficient than polling-based solutions
2. Flexibility: The tool supports numerous event types and filtering options to meet diverse monitoring needs
3. Scalability: With proper configuration and system tuning, `inotifywait` can handle large-scale monitoring requirements
4. Integration: The tool integrates well with shell scripts and system services for automated workflows
Next Steps
To further enhance your file monitoring capabilities, consider:
1. Exploring inotifywatch: The companion tool for collecting file system access statistics
2. Integration with Log Management: Combine `inotifywait` with tools like rsyslog or journald for centralized logging
3. Advanced Alerting: Implement integration with monitoring systems like Nagios, Zabbix, or Prometheus
4. Performance Monitoring: Use tools like `iostat` and `iotop` to monitor the performance impact of your monitoring setup
Additional Resources
For continued learning and troubleshooting:
- Linux inotify documentation: `/usr/share/doc/inotify-tools/`
- Kernel inotify interface: `man 7 inotify`
- System limits configuration: `/proc/sys/fs/inotify/`
By implementing the techniques and best practices outlined in this guide, you'll be well-equipped to create robust, efficient file monitoring solutions that meet your specific requirements while maintaining system performance and reliability.
Remember that effective file monitoring is not just about detecting changes—it's about responding to them appropriately and maintaining the security and integrity of your systems. Use `inotifywait` as part of a comprehensive monitoring and automation strategy to maximize its benefits.