How to configure DRBD on Linux

How to Configure DRBD on Linux Introduction Distributed Replicated Block Device (DRBD) is a powerful Linux kernel module that provides high-availability data replication by mirroring block devices over a network. Often referred to as "RAID 1 over the network," DRBD creates synchronized copies of data between two or more Linux systems, ensuring data availability even when hardware failures occur. This comprehensive guide will walk you through the complete process of configuring DRBD on Linux, from initial setup to advanced configurations. Whether you're building a high-availability cluster, implementing disaster recovery solutions, or simply ensuring data redundancy, this tutorial provides the knowledge and practical examples needed to successfully deploy DRBD in production environments. By the end of this guide, you'll understand how to install DRBD, configure resources, manage synchronization, handle failover scenarios, and troubleshoot common issues that may arise during implementation. Prerequisites and Requirements Before beginning the DRBD configuration process, ensure your environment meets the following requirements: Hardware Requirements - Two Linux servers (minimum) with network connectivity - Dedicated storage devices or partitions on each server for DRBD replication - Network interface for DRBD communication (dedicated network recommended) - Sufficient RAM (minimum 512MB, recommended 2GB or more) - Matching hardware configurations for optimal performance Software Requirements - Linux distribution with kernel version 2.6.33 or later - Root access on both servers - Network connectivity between nodes (low latency preferred) - Firewall configuration allowing DRBD traffic (typically port 7788) - Time synchronization between nodes (NTP recommended) Network Configuration Ensure both nodes can communicate reliably over the network. Configure dedicated network interfaces for DRBD traffic when possible to avoid interference with application traffic. Step-by-Step DRBD Configuration Step 1: Install DRBD Software Begin by installing DRBD on both nodes. The installation method varies depending on your Linux distribution. For Ubuntu/Debian Systems ```bash Update package repositories sudo apt update Install DRBD utilities and kernel module sudo apt install drbd-utils drbd-dkms Verify installation sudo modprobe drbd lsmod | grep drbd ``` For RHEL/CentOS/Rocky Linux ```bash Install EPEL repository (if not already installed) sudo dnf install epel-release Install DRBD packages sudo dnf install drbd90-utils kmod-drbd90 Load DRBD module sudo modprobe drbd echo drbd | sudo tee -a /etc/modules-load.d/drbd.conf ``` Step 2: Prepare Storage Devices Create dedicated partitions or use separate disks for DRBD on both nodes. The storage devices should be identical in size. ```bash Example: Create a partition using fdisk sudo fdisk /dev/sdb Create a 10GB partition (adjust size as needed) Follow fdisk prompts to create partition Verify partition creation lsblk /dev/sdb ``` Important: Do not format the partition or create filesystems yet. DRBD will manage the raw block device. Step 3: Configure DRBD Global Settings Create the global DRBD configuration file on both nodes: ```bash sudo nano /etc/drbd.d/global_common.conf ``` Add the following configuration: ```bash global { usage-count yes; udev-always-use-vnr; } common { handlers { pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f"; pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f"; local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f"; } startup { degr-wfc-timeout 120; } options { auto-promote yes; } disk { on-io-error detach; } net { cram-hmac-alg sha1; shared-secret "your-shared-secret-key"; after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary; after-sb-2pri call-pri-lost-after-sb; protocol C; } } ``` Step 4: Create DRBD Resource Configuration Create a resource configuration file for your DRBD setup: ```bash sudo nano /etc/drbd.d/r0.res ``` Add the following configuration (adjust hostnames, IP addresses, and device paths): ```bash resource r0 { on node1 { device /dev/drbd1; disk /dev/sdb1; address 192.168.1.10:7788; meta-disk internal; } on node2 { device /dev/drbd1; disk /dev/sdb1; address 192.168.1.11:7788; meta-disk internal; } } ``` Configuration Parameters Explained: - resource r0: Defines the DRBD resource name - device: Virtual DRBD device path - disk: Physical storage device path - address: IP address and port for DRBD communication - meta-disk internal: Stores metadata on the same device Step 5: Initialize DRBD Metadata Initialize DRBD metadata on both nodes: ```bash Create metadata sudo drbdadm create-md r0 Expected output should show successful metadata creation ``` Step 6: Start DRBD Service Enable and start DRBD on both nodes: ```bash Enable DRBD service sudo systemctl enable drbd Start DRBD service sudo systemctl start drbd Verify service status sudo systemctl status drbd ``` Step 7: Configure Primary Node Designate one node as the primary and perform initial synchronization: ```bash On the first node (node1), set as primary sudo drbdadm primary --force r0 Verify DRBD status sudo drbdadm status r0 ``` The output should show the synchronization process: ```bash r0 role:Primary disk:UpToDate node2 role:Secondary replication:SyncSource peer-disk:Inconsistent done:15.36 ``` Step 8: Create Filesystem and Mount Point Once synchronization completes, create a filesystem on the primary node: ```bash Create ext4 filesystem (only on primary node) sudo mkfs.ext4 /dev/drbd1 Create mount point sudo mkdir /mnt/drbd Mount the filesystem sudo mount /dev/drbd1 /mnt/drbd Verify mount df -h /mnt/drbd ``` Practical Examples and Use Cases Example 1: High-Availability File Server Configure DRBD for a high-availability file server setup: ```bash Create shared directory structure sudo mkdir -p /mnt/drbd/shared/{documents,backups} Set appropriate permissions sudo chown -R www-data:www-data /mnt/drbd/shared sudo chmod -R 755 /mnt/drbd/shared Test file creation sudo touch /mnt/drbd/shared/test-file.txt ``` Example 2: Database Replication Setup Configure DRBD for database high availability: ```bash Create database directory sudo mkdir -p /mnt/drbd/mysql Set MySQL ownership (if using MySQL) sudo chown -R mysql:mysql /mnt/drbd/mysql sudo chmod 750 /mnt/drbd/mysql Configure MySQL data directory in /etc/mysql/mysql.conf.d/mysqld.cnf datadir = /mnt/drbd/mysql ``` Example 3: Failover Testing Test manual failover between nodes: ```bash On primary node (node1) Unmount filesystem sudo umount /mnt/drbd Switch to secondary role sudo drbdadm secondary r0 On secondary node (node2) Promote to primary sudo drbdadm primary r0 Mount filesystem sudo mount /dev/drbd1 /mnt/drbd Verify data integrity ls -la /mnt/drbd/ ``` Advanced Configuration Options Multi-Master Configuration Configure DRBD for dual-primary mode (requires cluster filesystem): ```bash In /etc/drbd.d/r0.res, add to net section: net { allow-two-primaries yes; after-sb-0pri discard-least-changes; after-sb-1pri consensus; after-sb-2pri disconnect; } ``` Performance Tuning Optimize DRBD performance for your environment: ```bash Add to disk section in global_common.conf disk { resync-rate 100M; c-plan-ahead 20; c-fill-target 10M; c-max-rate 200M; } ``` Monitoring and Alerting Set up DRBD monitoring: ```bash Create monitoring script sudo nano /usr/local/bin/drbd-monitor.sh #!/bin/bash DRBD_STATUS=$(drbdadm status) if echo "$DRBD_STATUS" | grep -q "Inconsistent\|StandAlone"; then echo "DRBD Alert: $(date) - $DRBD_STATUS" | mail -s "DRBD Status Alert" admin@example.com fi ``` Common Issues and Troubleshooting Issue 1: Split-Brain Scenario Symptoms: Both nodes believe they are primary, causing data inconsistency. Solution: ```bash On the node with older data (check timestamps) sudo drbdadm secondary r0 sudo drbdadm disconnect r0 sudo drbdadm -- --discard-my-data connect r0 On the node with newer data sudo drbdadm connect r0 ``` Issue 2: Synchronization Stuck Symptoms: DRBD synchronization appears frozen or extremely slow. Diagnosis and Solution: ```bash Check current sync status sudo drbdadm status r0 Check for network issues ping -c 4 192.168.1.11 # Replace with peer IP Increase sync rate temporarily echo 200M | sudo tee /sys/block/drbd1/queue/resync_rate Check system resources iostat -x 1 5 ``` Issue 3: Connection Timeout Symptoms: DRBD shows "StandAlone" status and cannot connect to peer. Solution: ```bash Check firewall settings sudo iptables -L | grep 7788 sudo ufw status Allow DRBD traffic sudo ufw allow 7788 sudo firewall-cmd --permanent --add-port=7788/tcp sudo firewall-cmd --reload Restart DRBD connection sudo drbdadm disconnect r0 sudo drbdadm connect r0 ``` Issue 4: Metadata Corruption Symptoms: DRBD fails to start with metadata errors. Solution: ```bash Backup existing data if possible Recreate metadata (WARNING: This will destroy existing metadata) sudo drbdadm create-md --force r0 If data exists on one node, mark it as consistent sudo drbdadm -- --clear-bitmap new-current-uuid r0 ``` Best Practices and Professional Tips Security Best Practices 1. Use dedicated network interfaces for DRBD traffic to improve security and performance 2. Implement strong shared secrets in the configuration 3. Enable firewall rules specific to DRBD ports 4. Regular security audits of DRBD configurations ```bash Generate strong shared secret openssl rand -base64 32 ``` Performance Optimization 1. Use SSD storage for better synchronization performance 2. Configure appropriate buffer sizes based on network latency 3. Monitor network bandwidth usage during synchronization 4. Implement proper I/O scheduling for DRBD devices ```bash Set I/O scheduler for better DRBD performance echo deadline | sudo tee /sys/block/sdb/queue/scheduler ``` Monitoring and Maintenance 1. Regular status checks using automated scripts 2. Log monitoring for DRBD-related messages 3. Performance metrics collection for trend analysis 4. Backup verification of DRBD configurations ```bash Create daily status report sudo drbdadm status > /var/log/drbd-status-$(date +%Y%m%d).log ``` High Availability Integration 1. Integrate with cluster managers like Pacemaker 2. Implement proper fencing mechanisms to prevent split-brain 3. Configure automatic failover for critical services 4. Test failover procedures regularly Maintenance and Operations Regular Maintenance Tasks ```bash Weekly DRBD health check sudo drbdadm status sudo drbdadm dstate r0 Monitor synchronization performance cat /proc/drbd Check for errors in system logs sudo journalctl -u drbd -f ``` Backup Strategies ```bash Create consistent snapshot on secondary node sudo drbdadm secondary r0 sudo lvm snapshot /dev/drbd1 /dev/backup/drbd-snapshot sudo drbdadm primary r0 ``` Capacity Planning Monitor disk usage and plan for growth: ```bash Monitor DRBD device usage df -h /dev/drbd1 Check I/O statistics iostat -x /dev/drbd1 1 5 ``` Integration with Cluster Management Pacemaker Integration Configure DRBD with Pacemaker for automated failover: ```bash Create DRBD resource in Pacemaker sudo pcs resource create drbd_r0 ocf:linbit:drbd drbd_resource=r0 \ op monitor interval=60s sudo pcs resource master drbd_r0_master drbd_r0 \ master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 \ notify=true ``` Conclusion DRBD provides a robust solution for data replication and high availability in Linux environments. Through this comprehensive guide, you've learned how to install, configure, and manage DRBD systems effectively. Key takeaways include: - Proper planning is essential for successful DRBD deployment - Regular monitoring helps prevent issues before they impact operations - Testing failover procedures ensures reliability during actual emergencies - Performance tuning can significantly improve synchronization speed - Security considerations should be implemented from the beginning Next Steps After successfully configuring DRBD, consider these advanced topics: 1. Integrate with cluster management software like Pacemaker or Corosync 2. Implement automated monitoring and alerting systems 3. Explore three-node configurations for additional redundancy 4. Study disaster recovery procedures for geographic replication 5. Investigate DRBD Proxy for long-distance replication scenarios Remember that DRBD is a critical component of your infrastructure, so always test configurations thoroughly in non-production environments before deploying to production systems. Regular maintenance, monitoring, and testing will ensure your DRBD setup provides the high availability and data protection your organization requires. For ongoing support and advanced configurations, consult the official DRBD documentation and consider professional support services for mission-critical implementations.