How to set up high availability in Linux
How to Set Up High Availability in Linux
High availability (HA) is a critical requirement for modern enterprise systems, ensuring that services remain accessible even when individual components fail. In Linux environments, implementing high availability involves creating redundant systems, automated failover mechanisms, and continuous monitoring to minimize downtime and maintain service continuity. This comprehensive guide will walk you through the essential concepts, tools, and step-by-step procedures to establish robust high availability solutions in Linux.
Table of Contents
1. [Understanding High Availability](#understanding-high-availability)
2. [Prerequisites and Requirements](#prerequisites-and-requirements)
3. [High Availability Components](#high-availability-components)
4. [Setting Up Pacemaker and Corosync](#setting-up-pacemaker-and-corosync)
5. [Configuring Load Balancing](#configuring-load-balancing)
6. [Database High Availability](#database-high-availability)
7. [Storage Solutions for HA](#storage-solutions-for-ha)
8. [Monitoring and Alerting](#monitoring-and-alerting)
9. [Testing Your HA Setup](#testing-your-ha-setup)
10. [Troubleshooting Common Issues](#troubleshooting-common-issues)
11. [Best Practices](#best-practices)
12. [Conclusion](#conclusion)
Understanding High Availability
High availability refers to systems designed to remain operational for extended periods with minimal downtime. The goal is to achieve maximum uptime, typically measured in "nines" (99.9%, 99.99%, etc.). A properly configured HA system automatically detects failures and redirects traffic or services to healthy nodes without manual intervention.
Key Concepts
Redundancy: Multiple instances of critical components ensure that if one fails, others can take over seamlessly.
Failover: The automatic switching from a failed component to a standby component.
Load Balancing: Distributing workload across multiple servers to prevent any single point of failure.
Split-Brain Prevention: Mechanisms to prevent multiple nodes from simultaneously acting as the primary node.
Prerequisites and Requirements
Before implementing high availability in Linux, ensure you have the following prerequisites:
Hardware Requirements
- Minimum of two identical servers with similar specifications
- Dedicated network interfaces for cluster communication
- Shared storage (SAN, NAS, or distributed storage)
- Redundant network connections and switches
- Uninterruptible Power Supply (UPS) for both nodes
Software Requirements
- Linux distribution (CentOS, RHEL, Ubuntu, or SUSE)
- Root or sudo access on all cluster nodes
- Network Time Protocol (NTP) for time synchronization
- SSH key-based authentication between nodes
Network Configuration
```bash
Example network configuration for cluster nodes
Node 1 (Primary)
IP Address: 192.168.1.10
Cluster IP: 192.168.1.100 (Virtual IP)
Heartbeat Network: 10.0.0.10
Node 2 (Secondary)
IP Address: 192.168.1.11
Cluster IP: 192.168.1.100 (Virtual IP)
Heartbeat Network: 10.0.0.11
```
High Availability Components
Pacemaker
Pacemaker is the cluster resource manager that makes decisions about where and when to start, stop, or restart services based on the cluster's current state.
Corosync
Corosync provides the messaging layer for cluster communication, handling membership, messaging, and quorum services.
STONITH (Shoot The Other Node In The Head)
STONITH is a fencing mechanism that forcibly powers off or resets unresponsive nodes to prevent data corruption.
Resource Agents
Resource agents are scripts that manage specific services or applications, providing start, stop, and monitor operations.
Setting Up Pacemaker and Corosync
Step 1: Install Required Packages
On CentOS/RHEL 8:
```bash
Install high availability packages
sudo dnf install -y pcs pacemaker corosync fence-agents-all
Enable and start pcsd service
sudo systemctl enable pcsd
sudo systemctl start pcsd
```
On Ubuntu 20.04/22.04:
```bash
Install clustering packages
sudo apt update
sudo apt install -y pacemaker corosync crmsh fence-agents
Enable services
sudo systemctl enable pacemaker
sudo systemctl enable corosync
```
Step 2: Configure Cluster Authentication
Set up the hacluster user password on all nodes:
```bash
Set password for hacluster user (same on all nodes)
sudo passwd hacluster
```
Authenticate nodes (run on one node only):
```bash
Authenticate cluster nodes
sudo pcs host auth node1.example.com node2.example.com -u hacluster
```
Step 3: Create the Cluster
Initialize the cluster (run on one node only):
```bash
Create cluster named "webcluster"
sudo pcs cluster setup webcluster node1.example.com node2.example.com
Start cluster on all nodes
sudo pcs cluster start --all
Enable cluster to start on boot
sudo pcs cluster enable --all
```
Step 4: Configure Cluster Properties
```bash
Disable STONITH temporarily (enable in production)
sudo pcs property set stonith-enabled=false
Set cluster to ignore quorum (for 2-node clusters)
sudo pcs property set no-quorum-policy=ignore
Check cluster status
sudo pcs status
```
Step 5: Create Virtual IP Resource
```bash
Create virtual IP resource
sudo pcs resource create virtual_ip IPaddr2 \
ip=192.168.1.100 \
cidr_netmask=24 \
op monitor interval=30s
Verify resource status
sudo pcs status resources
```
Configuring Load Balancing
HAProxy Configuration
Install and configure HAProxy for load balancing:
```bash
Install HAProxy
sudo apt install -y haproxy # Ubuntu
sudo dnf install -y haproxy # CentOS/RHEL
```
Create HAProxy configuration file (`/etc/haproxy/haproxy.cfg`):
```bash
global
daemon
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
option httplog
frontend web_frontend
bind *:80
bind *:443 ssl crt /etc/ssl/certs/server.pem
default_backend web_servers
backend web_servers
balance roundrobin
option httpchk GET /health
server web1 192.168.1.20:80 check
server web2 192.168.1.21:80 check
server web3 192.168.1.22:80 check
listen stats
bind *:8404
stats enable
stats uri /stats
stats refresh 30s
```
Create HAProxy Resource in Pacemaker
```bash
Create HAProxy resource
sudo pcs resource create haproxy systemd:haproxy \
op monitor interval=30s \
op start timeout=60s \
op stop timeout=60s
Create resource group
sudo pcs resource group add webservice virtual_ip haproxy
Verify configuration
sudo pcs status
```
Database High Availability
MySQL/MariaDB Master-Slave Replication
Configure Master Server
Edit MySQL configuration (`/etc/mysql/mysql.conf.d/mysqld.cnf`):
```bash
[mysqld]
server-id = 1
log-bin = mysql-bin
binlog-do-db = production_db
```
Create replication user:
```sql
-- Connect to MySQL as root
CREATE USER 'replica'@'%' IDENTIFIED BY 'replica_password';
GRANT REPLICATION SLAVE ON . TO 'replica'@'%';
FLUSH PRIVILEGES;
-- Get master status
SHOW MASTER STATUS;
```
Configure Slave Server
Edit slave configuration:
```bash
[mysqld]
server-id = 2
relay-log = mysql-relay-bin
log-bin = mysql-bin
binlog-do-db = production_db
```
Configure replication:
```sql
-- Configure slave
CHANGE MASTER TO
MASTER_HOST='192.168.1.10',
MASTER_USER='replica',
MASTER_PASSWORD='replica_password',
MASTER_LOG_FILE='mysql-bin.000001',
MASTER_LOG_POS=154;
-- Start replication
START SLAVE;
-- Check slave status
SHOW SLAVE STATUS\G
```
MySQL Resource Agent Configuration
```bash
Create MySQL resource
sudo pcs resource create mysql mysql \
binary="/usr/bin/mysqld_safe" \
config="/etc/mysql/mysql.conf.d/mysqld.cnf" \
datadir="/var/lib/mysql" \
pid="/var/run/mysqld/mysqld.pid" \
socket="/var/run/mysqld/mysqld.sock" \
additional_parameters="--bind-address=0.0.0.0" \
op start timeout=60s \
op stop timeout=60s \
op monitor interval=20s timeout=30s
Add to resource group
sudo pcs resource group add dbservice mysql
```
Storage Solutions for HA
DRBD (Distributed Replicated Block Device)
DRBD creates a replicated block device across cluster nodes.
Install DRBD
```bash
Install DRBD packages
sudo apt install -y drbd-utils # Ubuntu
sudo dnf install -y drbd90-utils kmod-drbd90 # CentOS/RHEL
```
Configure DRBD Resource
Create DRBD configuration (`/etc/drbd.d/data.res`):
```bash
resource data {
protocol C;
meta-disk internal;
device /dev/drbd0;
syncer {
verify-alg sha1;
}
net {
allow-two-primaries;
}
on node1.example.com {
disk /dev/sdb1;
address 10.0.0.10:7788;
}
on node2.example.com {
disk /dev/sdb1;
address 10.0.0.11:7788;
}
}
```
Initialize DRBD
```bash
Create metadata
sudo drbdadm create-md data
Start DRBD
sudo systemctl enable drbd
sudo systemctl start drbd
Set primary node (run on primary only)
sudo drbdadm primary --force data
Create filesystem (primary node only)
sudo mkfs.ext4 /dev/drbd0
```
Create DRBD Resource in Pacemaker
```bash
Create DRBD resource
sudo pcs resource create drbd_data ocf:linbit:drbd \
drbd_resource=data \
op monitor interval=60s
Create master-slave resource
sudo pcs resource master drbd_data_clone drbd_data \
master-max=1 master-node-max=1 \
clone-max=2 clone-node-max=1 \
notify=true
Create filesystem resource
sudo pcs resource create shared_fs Filesystem \
device=/dev/drbd0 \
directory=/shared \
fstype=ext4 \
op monitor interval=20s
Create colocation constraint
sudo pcs constraint colocation add shared_fs with master drbd_data_clone
sudo pcs constraint order promote drbd_data_clone then start shared_fs
```
Monitoring and Alerting
Nagios Integration
Install Nagios plugins for cluster monitoring:
```bash
Install monitoring tools
sudo apt install -y nagios-plugins-basic monitoring-plugins-standard
Create custom check script
sudo tee /usr/local/bin/check_pacemaker.sh << 'EOF'
#!/bin/bash
STATUS=$(pcs status --full 2>/dev/null)
if echo "$STATUS" | grep -q "OFFLINE"; then
echo "CRITICAL: Node(s) offline in cluster"
exit 2
elif echo "$STATUS" | grep -q "Failed"; then
echo "WARNING: Failed resources detected"
exit 1
else
echo "OK: Cluster is healthy"
exit 0
fi
EOF
sudo chmod +x /usr/local/bin/check_pacemaker.sh
```
Prometheus and Grafana
Configure Prometheus to monitor cluster metrics:
```yaml
prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'pacemaker'
static_configs:
- targets: ['node1:9664', 'node2:9664']
scrape_interval: 30s
- job_name: 'haproxy'
static_configs:
- targets: ['192.168.1.100:8404']
metrics_path: /stats
params:
stats: [';csv']
```
Testing Your HA Setup
Failover Testing
Test Node Failure
```bash
Simulate node failure by stopping cluster services
sudo pcs cluster stop node1
Verify resources moved to surviving node
sudo pcs status
Restart failed node
sudo pcs cluster start node1
```
Test Service Failure
```bash
Stop specific resource
sudo pcs resource disable haproxy
Monitor resource status
watch 'sudo pcs status'
Re-enable resource
sudo pcs resource enable haproxy
```
Network Partition Testing
```bash
Block cluster communication (temporary)
sudo iptables -A INPUT -p udp --dport 5405 -j DROP
sudo iptables -A OUTPUT -p udp --dport 5405 -j DROP
Restore communication
sudo iptables -D INPUT -p udp --dport 5405 -j DROP
sudo iptables -D OUTPUT -p udp --dport 5405 -j DROP
```
Load Testing
Use Apache Bench to test load balancing:
```bash
Install Apache Bench
sudo apt install -y apache2-utils
Perform load test
ab -n 10000 -c 100 http://192.168.1.100/
Monitor HAProxy stats
curl http://192.168.1.100:8404/stats
```
Troubleshooting Common Issues
Split-Brain Scenarios
Symptoms: Multiple nodes claiming to be primary, conflicting resource states.
Solutions:
```bash
Check cluster status on all nodes
sudo pcs status
Clear split-brain condition
sudo pcs resource cleanup
If necessary, reset cluster
sudo pcs cluster stop --all
sudo pcs cluster start --all
```
Resource Startup Failures
Symptoms: Resources fail to start or keep restarting.
Diagnostic Commands:
```bash
Check resource failures
sudo pcs status --full
View resource logs
sudo pcs resource debug-start resource_name
Check system logs
sudo journalctl -u pacemaker -f
sudo journalctl -u corosync -f
```
Network Communication Issues
Symptoms: Nodes cannot communicate, frequent failovers.
Solutions:
```bash
Test cluster communication
sudo corosync-cfgtool -s
Check network connectivity
ping -c 4 node2.example.com
Verify firewall rules
sudo firewall-cmd --list-all
sudo ufw status # Ubuntu
```
Performance Issues
Symptoms: Slow failover times, resource delays.
Optimization:
```bash
Tune cluster timing
sudo pcs property set cluster-recheck-interval=60s
sudo pcs property set migration-threshold=3
Optimize resource monitoring
sudo pcs resource update resource_name op monitor interval=30s timeout=20s
```
Best Practices
Security Considerations
1. Enable STONITH: Always configure proper fencing mechanisms in production:
```bash
Configure IPMI fencing
sudo pcs stonith create node1_fence fence_ipmilan \
pcmk_host_list="node1.example.com" \
ipaddr="192.168.1.50" \
login="admin" \
passwd="password"
```
2. Secure Cluster Communication: Use encrypted communication between nodes:
```bash
Generate cluster key
sudo corosync-keygen
Copy to all nodes
sudo scp /etc/corosync/authkey node2:/etc/corosync/
```
3. Regular Security Updates: Keep all cluster components updated:
```bash
Automated security updates
sudo apt install -y unattended-upgrades # Ubuntu
sudo dnf install -y dnf-automatic # CentOS/RHEL
```
Monitoring and Maintenance
1. Regular Health Checks: Implement automated monitoring:
```bash
Cron job for cluster health checks
echo "/5 * root /usr/local/bin/check_pacemaker.sh" >> /etc/crontab
```
2. Log Rotation: Configure proper log management:
```bash
Configure logrotate for cluster logs
sudo tee /etc/logrotate.d/pacemaker << 'EOF'
/var/log/pacemaker/*.log {
daily
rotate 7
compress
delaycompress
missingok
notifempty
create 644 hacluster haclient
}
EOF
```
3. Backup Cluster Configuration: Regular configuration backups:
```bash
Backup cluster configuration
sudo pcs config backup cluster_backup_$(date +%Y%m%d)
Automated backup script
sudo tee /usr/local/bin/backup_cluster.sh << 'EOF'
#!/bin/bash
BACKUP_DIR="/backup/cluster"
mkdir -p $BACKUP_DIR
pcs config backup $BACKUP_DIR/cluster_$(date +%Y%m%d_%H%M%S)
find $BACKUP_DIR -name "cluster_*" -mtime +30 -delete
EOF
```
Capacity Planning
1. Resource Allocation: Ensure adequate resources for failover scenarios
2. Network Bandwidth: Plan for increased network traffic during synchronization
3. Storage Performance: Use high-performance storage for shared resources
4. Scalability: Design for horizontal scaling as demand grows
Documentation and Procedures
1. Runbooks: Create detailed operational procedures
2. Contact Information: Maintain updated emergency contact lists
3. Change Management: Document all configuration changes
4. Disaster Recovery: Develop and test disaster recovery procedures
Conclusion
Setting up high availability in Linux requires careful planning, proper implementation, and ongoing maintenance. This comprehensive guide has covered the essential components and procedures needed to create a robust HA environment using Pacemaker, Corosync, and related technologies.
Key takeaways for successful HA implementation:
- Start with proper planning: Understand your requirements and design accordingly
- Test thoroughly: Regular testing ensures your HA setup works when needed
- Monitor continuously: Implement comprehensive monitoring and alerting
- Document everything: Maintain detailed documentation for troubleshooting and maintenance
- Stay updated: Keep all components current with security updates and patches
Remember that high availability is not just about technology—it requires ongoing attention, regular testing, and continuous improvement. As your infrastructure grows and evolves, revisit your HA configuration to ensure it continues to meet your availability requirements.
The investment in high availability pays dividends in reduced downtime, improved user experience, and business continuity. By following the practices outlined in this guide, you'll be well-equipped to maintain highly available Linux systems that can withstand various failure scenarios while providing consistent service to your users.
For advanced implementations, consider exploring additional technologies such as container orchestration with Kubernetes, advanced storage solutions like Ceph, and cloud-native HA patterns that can further enhance your infrastructure's resilience and scalability.