How to assemble on boot → mdadm --assemble --scan
How to Assemble RAID Arrays on Boot Using mdadm --assemble --scan
Table of Contents
- [Introduction](#introduction)
- [Prerequisites](#prerequisites)
- [Understanding mdadm and RAID Assembly](#understanding-mdadm-and-raid-assembly)
- [The mdadm --assemble --scan Command](#the-mdadm---assemble---scan-command)
- [Configuring Automatic Assembly on Boot](#configuring-automatic-assembly-on-boot)
- [Step-by-Step Implementation Guide](#step-by-step-implementation-guide)
- [Practical Examples and Use Cases](#practical-examples-and-use-cases)
- [Advanced Configuration Options](#advanced-configuration-options)
- [Troubleshooting Common Issues](#troubleshooting-common-issues)
- [Best Practices and Professional Tips](#best-practices-and-professional-tips)
- [Monitoring and Maintenance](#monitoring-and-maintenance)
- [Security Considerations](#security-considerations)
- [Conclusion](#conclusion)
Introduction
RAID (Redundant Array of Independent Disks) arrays provide data redundancy, improved performance, and increased storage capacity for Linux systems. However, these arrays must be properly assembled during system boot to ensure your data remains accessible. The `mdadm --assemble --scan` command is the cornerstone of automatic RAID array assembly in Linux systems.
This comprehensive guide will teach you how to configure your Linux system to automatically assemble RAID arrays during boot using mdadm. You'll learn the underlying concepts, implementation steps, troubleshooting techniques, and best practices that system administrators and advanced users need to maintain reliable RAID configurations.
Whether you're managing a single-server setup or multiple enterprise systems, understanding automatic RAID assembly is crucial for maintaining system uptime and data availability. This article covers everything from basic configuration to advanced troubleshooting scenarios.
Prerequisites
Before proceeding with RAID array assembly configuration, ensure you have:
System Requirements
- Linux distribution with mdadm installed (most modern distributions include it)
- Root or sudo access to the system
- Existing RAID arrays created with mdadm
- Basic understanding of Linux system administration
- Familiarity with command-line interface
Software Dependencies
```bash
Check if mdadm is installed
which mdadm
Install mdadm on Debian/Ubuntu systems
sudo apt-get update
sudo apt-get install mdadm
Install mdadm on RHEL/CentOS/Fedora systems
sudo yum install mdadm
or for newer versions
sudo dnf install mdadm
```
Knowledge Prerequisites
- Understanding of RAID concepts (RAID 0, 1, 5, 6, 10)
- Basic Linux file system knowledge
- Familiarity with system boot processes
- Understanding of configuration file editing
Understanding mdadm and RAID Assembly
What is mdadm?
The `mdadm` utility is the standard tool for managing MD (Multiple Device) arrays in Linux. MD arrays are Linux's software RAID implementation, allowing you to combine multiple physical storage devices into logical arrays that provide redundancy, performance improvements, or both.
RAID Assembly Process
When a Linux system boots, it needs to:
1. Detect available storage devices
2. Identify which devices belong to RAID arrays
3. Assemble the arrays from their component devices
4. Mount the assembled arrays as specified in `/etc/fstab`
Without proper configuration, RAID arrays won't be automatically assembled, leaving your data inaccessible until manual intervention.
The Role of Metadata
RAID arrays store metadata on each member device that includes:
- Array UUID (Universally Unique Identifier)
- Array name and level
- Device roles and positions
- Creation time and version information
- Superblock format details
This metadata enables the `--scan` option to automatically discover and assemble arrays without explicit device specification.
The mdadm --assemble --scan Command
Command Syntax and Purpose
```bash
mdadm --assemble --scan [options]
```
The `--assemble --scan` combination tells mdadm to:
- Scan all available block devices for RAID metadata
- Identify devices that belong to the same arrays
- Assemble discovered arrays automatically
- Activate the arrays for system use
Key Command Options
| Option | Description | Example |
|--------|-------------|---------|
| `--scan` | Scan for arrays to assemble | `mdadm --assemble --scan` |
| `--run` | Force assembly even with missing devices | `mdadm --assemble --scan --run` |
| `--verbose` | Provide detailed output | `mdadm --assemble --scan --verbose` |
| `--config` | Specify configuration file | `mdadm --assemble --scan --config=/etc/mdadm.conf` |
| `--uuid` | Assemble specific array by UUID | `mdadm --assemble --scan --uuid=12345678:90abcdef` |
Manual vs. Automatic Assembly
Manual Assembly:
```bash
Assemble specific array with explicit devices
mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1
```
Automatic Assembly:
```bash
Let mdadm discover and assemble all arrays
mdadm --assemble --scan
```
The automatic approach is preferred for boot-time assembly as it doesn't require hardcoded device names that might change.
Configuring Automatic Assembly on Boot
The mdadm Configuration File
The primary configuration file for mdadm is typically located at:
- `/etc/mdadm/mdadm.conf` (Debian/Ubuntu)
- `/etc/mdadm.conf` (RHEL/CentOS/Fedora)
This file contains array definitions and assembly instructions that mdadm uses during boot.
Basic Configuration Structure
```bash
/etc/mdadm/mdadm.conf example
Array definitions
ARRAY /dev/md0 metadata=1.2 name=hostname:0 UUID=12345678:90abcdef:12345678:90abcdef
Email notifications
MAILADDR admin@example.com
Device scanning
DEVICE partitions
Program to run for events
PROGRAM /usr/share/mdadm/checkarray --cron %1
```
Generating Configuration Automatically
The safest way to create the configuration file is to generate it from existing arrays:
```bash
Generate configuration for all active arrays
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
Generate with more verbose output
mdadm --detail --scan --verbose >> /etc/mdadm/mdadm.conf
```
Step-by-Step Implementation Guide
Step 1: Verify Existing RAID Arrays
Before configuring automatic assembly, identify your current RAID arrays:
```bash
List all active MD arrays
cat /proc/mdstat
Get detailed information about each array
mdadm --detail /dev/md0
mdadm --detail /dev/md1
```
Example output:
```
Personalities : [raid1] [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid10]
md0 : active raid1 sdb1[1] sda1[0]
1048512 blocks super 1.2 [2/2] [UU]
md1 : active raid5 sdd1[3] sdc1[2] sdb2[1] sda2[0]
3145728 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
```
Step 2: Create or Update Configuration File
Create a backup of existing configuration:
```bash
sudo cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf.backup
```
Generate new configuration:
```bash
Clear existing array definitions (keep other settings)
sudo sed -i '/^ARRAY/d' /etc/mdadm/mdadm.conf
Add current array configurations
sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf
```
Step 3: Verify Configuration File
Review the generated configuration:
```bash
sudo cat /etc/mdadm/mdadm.conf
```
Expected content:
```bash
DEVICE partitions
ARRAY /dev/md0 metadata=1.2 name=server:0 UUID=a1b2c3d4:e5f6a7b8:c9d0e1f2:a3b4c5d6
ARRAY /dev/md1 metadata=1.2 name=server:1 UUID=f1e2d3c4:b5a69788:c9d0e1f2:a3b4c5d6
MAILADDR root@localhost
```
Step 4: Update Initial RAM Filesystem
Most modern Linux distributions require updating the initial RAM filesystem (initramfs/initrd) to include RAID configuration:
For Debian/Ubuntu:
```bash
sudo update-initramfs -u
```
For RHEL/CentOS/Fedora:
```bash
sudo dracut -f
```
For SUSE:
```bash
sudo mkinitrd
```
Step 5: Configure Boot Services
Ensure mdadm service is enabled:
For systemd-based systems:
```bash
Enable mdmonitor service
sudo systemctl enable mdmonitor
Check service status
sudo systemctl status mdmonitor
```
For SysV init systems:
```bash
Enable mdadm service
sudo chkconfig mdadm on
```
Step 6: Test Configuration
Test the configuration without rebooting:
```bash
Stop arrays (WARNING: Ensure data is not in use)
sudo mdadm --stop /dev/md0
sudo mdadm --stop /dev/md1
Test automatic assembly
sudo mdadm --assemble --scan --verbose
Verify arrays are active
cat /proc/mdstat
```
Step 7: Reboot and Verify
Perform a system reboot to test automatic assembly:
```bash
sudo reboot
```
After reboot, verify arrays assembled correctly:
```bash
Check array status
cat /proc/mdstat
Verify all arrays are active and healthy
sudo mdadm --detail /dev/md0
sudo mdadm --detail /dev/md1
```
Practical Examples and Use Cases
Example 1: Simple RAID 1 Mirror Setup
Scenario: Two-disk RAID 1 array for system redundancy
Configuration:
```bash
Create RAID 1 array
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1
Generate configuration entry
sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf
Update initramfs
sudo update-initramfs -u
```
Resulting configuration:
```bash
ARRAY /dev/md0 metadata=1.2 name=server:0 UUID=12345678:90abcdef:12345678:90abcdef
```
Example 2: Multi-Array Server Setup
Scenario: Server with separate RAID arrays for OS and data
Arrays:
- `/dev/md0`: RAID 1 for operating system (2 × 120GB SSDs)
- `/dev/md1`: RAID 5 for data storage (4 × 2TB HDDs)
Configuration process:
```bash
Create OS RAID 1 array
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1
Create data RAID 5 array
sudo mdadm --create /dev/md1 --level=5 --raid-devices=4 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
Generate complete configuration
sudo mdadm --detail --scan > /etc/mdadm/mdadm.conf
Add additional settings
echo "DEVICE partitions" >> /etc/mdadm/mdadm.conf
echo "MAILADDR admin@company.com" >> /etc/mdadm/mdadm.conf
```
Example 3: Hot Spare Configuration
Scenario: RAID 5 array with hot spare for automatic recovery
Setup:
```bash
Create RAID 5 with hot spare
sudo mdadm --create /dev/md0 --level=5 --raid-devices=3 --spare-devices=1 \
/dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
Configuration includes spare device information
sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf
```
Configuration file entry:
```bash
ARRAY /dev/md0 metadata=1.2 name=server:0 UUID=12345678:90abcdef:12345678:90abcdef
DEVICE partitions
MAILADDR admin@company.com
```
Advanced Configuration Options
Custom Assembly Rules
You can create more specific assembly rules for complex scenarios:
```bash
Assemble only specific arrays
ARRAY /dev/md/boot UUID=12345678:90abcdef:12345678:90abcdef auto=yes
Assemble with custom device name
ARRAY /dev/md/data UUID=87654321:fedcba09:87654321:fedcba09 name=server:data
Assemble but don't auto-start
ARRAY /dev/md1 UUID=11111111:22222222:33333333:44444444 auto=no
```
Email Notification Setup
Configure email alerts for RAID events:
```bash
Basic email configuration
MAILADDR root@localhost
MAILADDR admin@company.com
Advanced email settings with program execution
PROGRAM /usr/local/bin/raid-alert.sh
```
Custom alert script example:
```bash
#!/bin/bash
/usr/local/bin/raid-alert.sh
EVENT="$1"
DEVICE="$2"
echo "RAID Event: $EVENT on device $DEVICE" | \
mail -s "RAID Alert on $(hostname)" admin@company.com
```
Performance Tuning Options
Optimize assembly performance for large arrays:
```bash
Increase assembly timeout
echo 'TIMEOUT=300' >> /etc/mdadm/mdadm.conf
Optimize for SSD arrays
echo 'AUTO +imsm +1.x -all' >> /etc/mdadm/mdadm.conf
```
Troubleshooting Common Issues
Issue 1: Arrays Not Assembling on Boot
Symptoms:
- Arrays show as inactive after boot
- Manual assembly works with `mdadm --assemble --scan`
- File systems not mounting
Diagnostic steps:
```bash
Check if arrays are detected but not assembled
sudo mdadm --examine --scan
Check configuration file syntax
sudo mdadm --config-file=/etc/mdadm/mdadm.conf --assemble --scan --test
Check boot logs for errors
sudo journalctl -b | grep -i mdadm
sudo dmesg | grep -i raid
```
Solutions:
1. Update initramfs:
```bash
sudo update-initramfs -u -k all
```
2. Fix configuration file:
```bash
sudo mdadm --detail --scan > /etc/mdadm/mdadm.conf.new
sudo mv /etc/mdadm/mdadm.conf.new /etc/mdadm/mdadm.conf
```
3. Check device naming:
```bash
# Use UUID instead of device names
sudo blkid /dev/md*
```
Issue 2: UUID Conflicts
Symptoms:
- Arrays assemble with wrong device names
- Multiple arrays claim same UUID
- Assembly fails with UUID errors
Diagnostic steps:
```bash
Check for duplicate UUIDs
sudo mdadm --examine /dev/sd* | grep -i uuid | sort
Examine specific devices
sudo mdadm --examine /dev/sda1
```
Solutions:
1. Regenerate UUIDs:
```bash
sudo mdadm --stop /dev/md0
sudo mdadm --assemble /dev/md0 --update=uuid /dev/sda1 /dev/sdb1
```
2. Update configuration:
```bash
sudo mdadm --detail --scan > /etc/mdadm/mdadm.conf
sudo update-initramfs -u
```
Issue 3: Missing or Failed Devices
Symptoms:
- Arrays won't assemble due to missing devices
- Degraded arrays won't start automatically
- Boot process hangs waiting for devices
Diagnostic steps:
```bash
Check array status
cat /proc/mdstat
Examine individual devices
sudo mdadm --examine /dev/sda1 /dev/sdb1 /dev/sdc1
Check for device failures
sudo dmesg | grep -i "I/O error"
```
Solutions:
1. Force assembly with missing devices:
```bash
sudo mdadm --assemble /dev/md0 --run --force /dev/sda1 /dev/sdb1
```
2. Configure automatic degraded startup:
```bash
# Add to /etc/mdadm/mdadm.conf
echo "AUTO -all +imsm +1.x" >> /etc/mdadm/mdadm.conf
```
3. Replace failed devices:
```bash
# Add new device to array
sudo mdadm --add /dev/md0 /dev/sdd1
# Remove failed device
sudo mdadm --remove /dev/md0 /dev/sdb1
```
Issue 4: Slow Boot Times
Symptoms:
- System takes long time to boot
- Boot process hangs at "Assembling RAID arrays"
- Timeout errors in boot logs
Solutions:
1. Increase timeout values:
```bash
# Edit /etc/default/mdadm
INITRDSTART='all'
AUTOSTART=true
```
2. Optimize device detection:
```bash
# Specify exact devices instead of 'partitions'
DEVICE /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
```
3. Use parallel assembly:
```bash
# Add to kernel parameters
raid=noautodetect
```
Best Practices and Professional Tips
Configuration Management
1. Always backup configuration files:
```bash
sudo cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf.$(date +%Y%m%d)
```
2. Use version control for configuration:
```bash
cd /etc
sudo git init
sudo git add mdadm/
sudo git commit -m "Initial RAID configuration"
```
3. Document your RAID setup:
```bash
# Create documentation file
sudo mdadm --detail --scan > /root/raid-documentation.txt
sudo mdadm --detail /dev/md* >> /root/raid-documentation.txt
```
Monitoring and Alerting
1. Set up comprehensive monitoring:
```bash
# Enable mdmonitor service
sudo systemctl enable mdmonitor
# Configure email alerts
echo "MAILADDR admin@company.com" >> /etc/mdadm/mdadm.conf
```
2. Regular health checks:
```bash
#!/bin/bash
# /usr/local/bin/raid-health-check.sh
for array in $(cat /proc/mdstat | grep ^md | cut -d: -f1); do
status=$(mdadm --detail /dev/$array | grep "State :" | cut -d: -f2)
echo "Array $array: $status"
done
```
3. Automated testing:
```bash
# Add to crontab for monthly consistency checks
0 2 1 /usr/share/mdadm/checkarray --cron --all --quiet
```
Security Considerations
1. Protect configuration files:
```bash
sudo chmod 600 /etc/mdadm/mdadm.conf
sudo chown root:root /etc/mdadm/mdadm.conf
```
2. Use encrypted arrays when needed:
```bash
# Create encrypted RAID array
sudo cryptsetup luksFormat /dev/md0
sudo cryptsetup luksOpen /dev/md0 encrypted_raid
```
3. Regular backup verification:
```bash
# Test array assembly in read-only mode
sudo mdadm --assemble --readonly /dev/md99 /dev/sda1 /dev/sdb1
```
Performance Optimization
1. Optimize chunk sizes for workload:
```bash
# For database workloads (small random I/O)
sudo mdadm --create /dev/md0 --level=5 --chunk=64 --raid-devices=4 /dev/sd[abcd]1
# For streaming workloads (large sequential I/O)
sudo mdadm --create /dev/md0 --level=5 --chunk=512 --raid-devices=4 /dev/sd[abcd]1
```
2. Configure appropriate read-ahead:
```bash
# Set read-ahead for RAID arrays
echo 'ACTION=="add|change", KERNEL=="md*", ATTR{bdi/read_ahead_kb}="4096"' > /etc/udev/rules.d/99-raid-readahead.rules
```
Disaster Recovery Planning
1. Create rescue documentation:
```bash
# Document exact recreation commands
echo "mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1" > /root/raid-recreation.txt
```
2. Test recovery procedures:
```bash
# Simulate failure and recovery
sudo mdadm --fail /dev/md0 /dev/sdb1
sudo mdadm --remove /dev/md0 /dev/sdb1
sudo mdadm --add /dev/md0 /dev/sdc1
```
Monitoring and Maintenance
Automated Monitoring Setup
Create a comprehensive monitoring solution:
```bash
#!/bin/bash
/usr/local/bin/mdadm-monitor.sh
LOGFILE="/var/log/mdadm-monitor.log"
EMAIL="admin@company.com"
Function to log messages
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S'): $1" >> $LOGFILE
}
Check array status
check_arrays() {
for array in $(awk '/^md/ {print $1}' /proc/mdstat); do
status=$(mdadm --detail /dev/$array 2>/dev/null | grep "State :" | awk '{print $3}')
if [[ "$status" != "clean" && "$status" != "active" ]]; then
log_message "WARNING: Array $array status is $status"
echo "RAID array $array has status: $status" | mail -s "RAID Alert" $EMAIL
fi
done
}
Check for failed devices
check_failed_devices() {
failed=$(grep -c "F" /proc/mdstat)
if [ $failed -gt 0 ]; then
log_message "ERROR: $failed failed devices detected"
echo "RAID arrays have $failed failed devices" | mail -s "CRITICAL: RAID Failure" $EMAIL
fi
}
Main execution
check_arrays
check_failed_devices
log_message "Monitor check completed"
```
Regular Maintenance Tasks
1. Monthly consistency checks:
```bash
# Add to root's crontab
0 3 1 /usr/share/mdadm/checkarray --cron --all --quiet
```
2. Weekly status reports:
```bash
#!/bin/bash
# /usr/local/bin/weekly-raid-report.sh
{
echo "Weekly RAID Status Report for $(hostname)"
echo "Generated on $(date)"
echo "=================================="
echo
cat /proc/mdstat
echo
for array in $(awk '/^md/ {print $1}' /proc/mdstat); do
echo "Details for /dev/$array:"
mdadm --detail /dev/$array
echo
done
} | mail -s "Weekly RAID Report" admin@company.com
```
Security Considerations
Access Control
1. Limit mdadm access:
```bash
# Create mdadm group
sudo groupadd mdadm-admin
# Add users to group
sudo usermod -a -G mdadm-admin username
# Configure sudo access
echo "%mdadm-admin ALL=(root) /sbin/mdadm" >> /etc/sudoers.d/mdadm
```
2. Audit RAID operations:
```bash
# Enable auditd for mdadm
echo "-w /sbin/mdadm -p x -k raid_commands" >> /etc/audit/rules.d/raid.rules
sudo service auditd restart
```
Data Protection
1. Implement encryption:
```bash
# Create encrypted RAID
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1
sudo cryptsetup luksFormat /dev/md0
sudo cryptsetup luksOpen /dev/md0 secure_storage
```
2. Secure configuration files:
```bash
# Set restrictive permissions
sudo chmod 640 /etc/mdadm/mdadm.conf
sudo chown root:root /etc/mdadm/mdadm.conf
# Create backup with encryption
sudo tar -czf - /etc/mdadm/ | gpg --cipher-algo AES256 --compress-algo 1 \
--symmetric --output /root/mdadm-config-backup.tar.gz.gpg
```
Conclusion
Configuring automatic RAID array assembly using `mdadm --assemble --scan` is essential for maintaining reliable Linux systems with software RAID. This comprehensive guide has covered all aspects of implementation, from basic setup to advanced troubleshooting and security considerations.
Key Takeaways
1. Proper configuration is critical: Always maintain accurate `/etc/mdadm/mdadm.conf` files and keep initramfs updated.
2. Testing is essential: Regularly test your RAID assembly configuration through controlled reboots and failure simulations.
3. Monitoring prevents disasters: Implement comprehensive monitoring and alerting to detect issues before they become critical.
4. Documentation saves time: Maintain detailed documentation of your RAID configurations and recovery procedures.
5. Security matters: Protect your RAID configuration and consider encryption for sensitive data.
Next Steps
After implementing automatic RAID assembly, consider these additional improvements:
- Set up centralized logging for RAID events across multiple systems
- Implement automated backup verification procedures
- Create disaster recovery runbooks with step-by-step procedures
- Consider migrating to more advanced storage solutions like LVM on RAID
- Explore hardware RAID solutions for performance-critical applications
Final Recommendations
Remember that software RAID is just one component of a comprehensive data protection strategy. Always maintain regular backups, test recovery procedures, and stay informed about best practices in storage management. The time invested in proper RAID configuration and monitoring will pay dividends in system reliability and data protection.
By following the guidance in this article, you'll have a robust, automatically-assembling RAID configuration that provides both performance and reliability for your Linux systems. Regular maintenance and monitoring will ensure your RAID arrays continue to protect your valuable data for years to come.