How to build a failover cluster in Linux

How to Build a Failover Cluster in Linux High availability is crucial for mission-critical applications and services in today's digital landscape. A failover cluster provides redundancy and automatic recovery capabilities, ensuring minimal downtime when hardware or software failures occur. This comprehensive guide will walk you through building a robust failover cluster in Linux, covering everything from initial setup to advanced configuration and troubleshooting. Table of Contents 1. [Understanding Failover Clusters](#understanding-failover-clusters) 2. [Prerequisites and Requirements](#prerequisites-and-requirements) 3. [Cluster Architecture Overview](#cluster-architecture-overview) 4. [Step-by-Step Implementation](#step-by-step-implementation) 5. [Configuration Examples](#configuration-examples) 6. [Testing and Validation](#testing-and-validation) 7. [Troubleshooting Common Issues](#troubleshooting-common-issues) 8. [Best Practices and Optimization](#best-practices-and-optimization) 9. [Advanced Topics](#advanced-topics) 10. [Conclusion and Next Steps](#conclusion-and-next-steps) Understanding Failover Clusters A failover cluster is a group of independent servers working together to increase application and service availability. When one server (node) fails, another node automatically takes over its workload with minimal service interruption. This process, called failover, ensures business continuity and reduces the impact of hardware or software failures. Key Components Cluster Nodes: Physical or virtual servers that participate in the cluster. Typically, you'll have at least two nodes for basic redundancy. Shared Storage: Centralized storage accessible by all cluster nodes, ensuring data consistency and availability during failovers. Cluster Software: Applications like Pacemaker, Corosync, and DRBD that manage cluster operations, resource monitoring, and failover decisions. Fencing Mechanisms: Systems that prevent split-brain scenarios by isolating failed or unresponsive nodes. Virtual IP (VIP): A floating IP address that moves between nodes during failover, maintaining client connectivity. Prerequisites and Requirements Before building your Linux failover cluster, ensure you have the following prerequisites in place: Hardware Requirements - Minimum two Linux servers (physical or virtual machines) - Identical hardware configuration recommended for optimal performance - Multiple network interfaces for redundant communication paths - Shared storage system (SAN, NAS, or distributed storage) - Uninterruptible Power Supply (UPS) for power redundancy Software Requirements - Linux Distribution: CentOS/RHEL 7+, Ubuntu 18.04+, or SUSE Linux Enterprise - Cluster Stack: Pacemaker and Corosync - Storage Replication: DRBD (if using local storage replication) - Network Time Protocol (NTP): For time synchronization - SSH Access: Passwordless SSH between cluster nodes Network Configuration - Dedicated cluster network: Separate network segment for cluster communication - Heartbeat links: Multiple network paths for redundancy - Firewall configuration: Proper ports opened for cluster communication - DNS resolution: Consistent hostname resolution across all nodes Cluster Architecture Overview Two-Node Active-Passive Cluster This is the most common failover cluster configuration, where one node actively serves requests while the other remains on standby, ready to take over if needed. ``` [Client] --> [Virtual IP] --> [Active Node] [Passive Node] (Standby) | [Shared Storage] ``` Multi-Node Active-Active Cluster In this configuration, multiple nodes can simultaneously serve different services or share the load, providing both high availability and load distribution. Step-by-Step Implementation Step 1: Prepare the Environment First, ensure both nodes have consistent hostnames and network configuration: ```bash On Node 1 hostnamectl set-hostname cluster-node1.example.com On Node 2 hostnamectl set-hostname cluster-node2.example.com ``` Update the `/etc/hosts` file on both nodes: ```bash cat >> /etc/hosts << EOF 192.168.1.10 cluster-node1.example.com cluster-node1 192.168.1.11 cluster-node2.example.com cluster-node2 192.168.1.12 cluster-vip.example.com cluster-vip EOF ``` Step 2: Install Required Packages On CentOS/RHEL systems: ```bash Install cluster packages yum install -y pcs pacemaker corosync fence-agents-all Start and enable pcsd service systemctl start pcsd systemctl enable pcsd ``` On Ubuntu systems: ```bash Install cluster packages apt-get update apt-get install -y pacemaker corosync crmsh fence-agents Start and enable services systemctl start pacemaker systemctl enable pacemaker systemctl start corosync systemctl enable corosync ``` Step 3: Configure Firewall Rules Open necessary ports for cluster communication: ```bash For CentOS/RHEL with firewalld firewall-cmd --permanent --add-service=high-availability firewall-cmd --reload For Ubuntu with ufw ufw allow 2224/tcp ufw allow 3121/tcp ufw allow 21064/tcp ufw allow 5405/udp ``` Step 4: Set Up Authentication Create a cluster user and configure authentication: ```bash Set password for hacluster user (same password on all nodes) passwd hacluster Authenticate nodes (run from one node) pcs host auth cluster-node1 cluster-node2 -u hacluster -p your_password ``` Step 5: Create the Cluster Initialize the cluster configuration: ```bash Create cluster (run from one node) pcs cluster setup my-cluster cluster-node1 cluster-node2 Start cluster services on all nodes pcs cluster start --all Enable cluster services to start at boot pcs cluster enable --all ``` Step 6: Configure Cluster Properties Set basic cluster properties: ```bash Disable STONITH for testing (enable in production) pcs property set stonith-enabled=false Set no-quorum policy for 2-node cluster pcs property set no-quorum-policy=ignore Configure cluster check interval pcs property set cluster-recheck-interval=60s ``` Step 7: Create Cluster Resources Configure Virtual IP Resource ```bash Create virtual IP resource pcs resource create cluster-vip IPaddr2 \ ip=192.168.1.12 \ cidr_netmask=24 \ op monitor interval=30s ``` Configure Service Resource (Example: Apache Web Server) ```bash Install Apache on both nodes yum install -y httpd # CentOS/RHEL or apt-get install -y apache2 # Ubuntu Create Apache resource pcs resource create webserver apache \ configfile="/etc/httpd/conf/httpd.conf" \ op monitor interval=30s ``` Create Resource Group Group related resources to ensure they run on the same node: ```bash Create resource group pcs resource group add web-service cluster-vip webserver ``` Step 8: Configure Shared Storage (DRBD Example) If using DRBD for storage replication: ```bash Install DRBD yum install -y drbd84-utils kmod-drbd84 # CentOS/RHEL or apt-get install -y drbd-utils # Ubuntu Configure DRBD resource cat > /etc/drbd.d/data.res << EOF resource data { protocol C; device /dev/drbd0; disk /dev/sdb1; meta-disk internal; on cluster-node1 { address 192.168.1.10:7788; } on cluster-node2 { address 192.168.1.11:7788; } } EOF Initialize DRBD drbdadm create-md data drbdadm up data Make one node primary and create filesystem drbdadm primary data --force mkfs.ext4 /dev/drbd0 ``` Configuration Examples Example 1: Database Cluster Configuration Here's a complete example for setting up a MySQL database cluster: ```bash Install MySQL on both nodes yum install -y mysql-server Create MySQL resource pcs resource create mysql-db mysql \ binary="/usr/libexec/mysqld" \ config="/etc/my.cnf" \ datadir="/var/lib/mysql" \ user="mysql" \ group="mysql" \ op start timeout=60s \ op stop timeout=60s \ op monitor interval=20s timeout=30s Create filesystem resource for shared storage pcs resource create mysql-fs Filesystem \ device="/dev/drbd0" \ directory="/var/lib/mysql" \ fstype="ext4" \ op monitor interval=20s Create resource group pcs resource group add mysql-service cluster-vip mysql-fs mysql-db Add ordering constraint pcs constraint order mysql-fs then mysql-db ``` Example 2: Load Balancer Configuration Configuration for a high-availability load balancer using HAProxy: ```bash Install HAProxy yum install -y haproxy Create HAProxy resource pcs resource create loadbalancer haproxy \ conffile="/etc/haproxy/haproxy.cfg" \ op monitor interval=30s Group with VIP pcs resource group add lb-service cluster-vip loadbalancer ``` Testing and Validation Verify Cluster Status ```bash Check overall cluster status pcs status Check cluster configuration pcs config View resource status pcs resource show ``` Test Failover Scenarios Manual Failover Testing ```bash Move resource to another node pcs resource move web-service cluster-node2 Clear movement constraint pcs resource clear web-service ``` Simulate Node Failure ```bash Stop cluster services on active node pcs cluster stop cluster-node1 Verify failover occurred pcs status ``` Performance Testing ```bash Test application response during failover while true; do curl -s http://cluster-vip.example.com > /dev/null if [ $? -ne 0 ]; then echo "$(date): Service unavailable" else echo "$(date): Service available" fi sleep 1 done ``` Troubleshooting Common Issues Split-Brain Prevention Split-brain occurs when cluster nodes cannot communicate but both remain active. Implement proper fencing: ```bash Configure STONITH device (example for IPMI) pcs stonith create node1-fence fence_ipmilan \ pcmk_host_list="cluster-node1" \ ipaddr="192.168.1.20" \ login="admin" \ passwd="password" Enable STONITH pcs property set stonith-enabled=true ``` Resource Startup Failures Check resource logs and configuration: ```bash View cluster logs journalctl -f -u pacemaker journalctl -f -u corosync Check resource configuration pcs resource show resource-name Debug resource issues pcs resource debug-start resource-name ``` Network Communication Issues Verify cluster communication: ```bash Check corosync status corosync-cfgtool -s Test multicast connectivity corosync-quorumtool -l Verify network configuration ip addr show ``` Common Error Solutions Error: "Resource is not running anywhere" ```bash Check resource constraints pcs constraint show Clear failed actions pcs resource cleanup resource-name ``` Error: "Quorum lost" ```bash For 2-node clusters, set no-quorum policy pcs property set no-quorum-policy=ignore ``` Best Practices and Optimization Security Hardening 1. Enable STONITH: Always use fencing in production environments 2. Secure Communication: Use encrypted cluster communication 3. Access Control: Implement proper user authentication and authorization 4. Network Segmentation: Use dedicated networks for cluster traffic Performance Optimization 1. Resource Monitoring: Tune monitoring intervals based on application requirements 2. Network Optimization: Use bonded interfaces for redundancy and performance 3. Storage Performance: Ensure shared storage meets performance requirements 4. Resource Limits: Configure appropriate timeout values Monitoring and Alerting ```bash Set up cluster monitoring with Nagios/Zabbix Example monitoring script #!/bin/bash CLUSTER_STATUS=$(pcs status | grep -c "Online:") if [ $CLUSTER_STATUS -lt 2 ]; then echo "CRITICAL: Cluster node offline" exit 2 fi echo "OK: All cluster nodes online" exit 0 ``` Backup and Recovery 1. Configuration Backup: Regularly backup cluster configuration ```bash pcs config backup cluster-backup.tar.bz2 ``` 2. Data Backup: Implement regular backups of shared storage 3. Disaster Recovery: Document and test recovery procedures Advanced Topics Multi-Site Clustering For geographic redundancy, consider multi-site clustering: ```bash Configure site-aware clustering pcs constraint location web-service prefers cluster-node1=100 pcs constraint location web-service prefers cluster-node2=50 ``` Integration with Cloud Platforms Cloud-specific considerations: - AWS: Use Elastic IPs and Route 53 for failover - Azure: Implement Azure Load Balancer integration - GCP: Use Google Cloud Load Balancing Container Orchestration Integration Integrate with Kubernetes or Docker Swarm for containerized applications: ```bash Example Kubernetes integration pcs resource create k8s-master systemd:kubelet \ op monitor interval=30s ``` Automated Deployment Use configuration management tools: ```yaml Ansible playbook example - name: Configure cluster hosts: cluster_nodes tasks: - name: Install cluster packages package: name: "{{ cluster_packages }}" state: present ``` Maintenance and Updates Regular Maintenance Tasks 1. Log Rotation: Configure log rotation for cluster logs 2. Resource Cleanup: Regularly clean up failed resource actions 3. Performance Monitoring: Monitor cluster performance metrics 4. Security Updates: Keep cluster software updated Upgrade Procedures ```bash Rolling upgrade procedure 1. Put node in standby pcs node standby cluster-node1 2. Update packages yum update pacemaker corosync 3. Bring node back online pcs node unstandby cluster-node1 4. Repeat for other nodes ``` Conclusion and Next Steps Building a failover cluster in Linux requires careful planning, proper configuration, and ongoing maintenance. This guide has covered the essential steps from initial setup through advanced configuration and troubleshooting. Key takeaways include: 1. Proper Planning: Understand your requirements and design appropriate architecture 2. Comprehensive Testing: Thoroughly test all failover scenarios 3. Security First: Implement proper security measures including fencing 4. Monitoring: Set up comprehensive monitoring and alerting 5. Documentation: Maintain detailed documentation of your cluster configuration Next Steps 1. Production Deployment: Implement the cluster in a test environment first 2. Performance Tuning: Optimize configuration based on your specific workload 3. Disaster Recovery: Develop and test disaster recovery procedures 4. Training: Ensure your team is trained on cluster operations and troubleshooting 5. Automation: Implement automation for routine maintenance tasks Additional Resources - Official Documentation: Consult Pacemaker and Corosync documentation - Community Forums: Participate in Linux clustering communities - Training: Consider formal training on Linux clustering technologies - Certification: Pursue relevant certifications for validation of skills By following this comprehensive guide, you'll have a solid foundation for implementing and managing Linux failover clusters that provide high availability for your critical applications and services. Remember that clustering is an ongoing process that requires regular attention, monitoring, and maintenance to ensure optimal performance and reliability.