How to Configure LVS Load Balancing in Linux
Linux Virtual Server (LVS) is a powerful, open-source load balancing solution that enables high-availability and scalable network services. Built into the Linux kernel, LVS provides exceptional performance for distributing incoming network requests across multiple backend servers. This comprehensive guide will walk you through the complete process of configuring LVS load balancing, from basic setup to advanced configurations and troubleshooting.
Table of Contents
1. [Understanding LVS Load Balancing](#understanding-lvs-load-balancing)
2. [Prerequisites and Requirements](#prerequisites-and-requirements)
3. [LVS Architecture and Components](#lvs-architecture-and-components)
4. [Installation and Initial Setup](#installation-and-initial-setup)
5. [Configuring LVS Director](#configuring-lvs-director)
6. [Setting Up Real Servers](#setting-up-real-servers)
7. [Load Balancing Methods](#load-balancing-methods)
8. [Practical Configuration Examples](#practical-configuration-examples)
9. [Monitoring and Management](#monitoring-and-management)
10. [Troubleshooting Common Issues](#troubleshooting-common-issues)
11. [Best Practices and Optimization](#best-practices-and-optimization)
12. [Advanced Configurations](#advanced-configurations)
13. [Conclusion](#conclusion)
Understanding LVS Load Balancing
Linux Virtual Server (LVS) is a kernel-level load balancing solution that operates at Layer 4 (transport layer) of the OSI model. Unlike application-level load balancers, LVS processes packets at the network level, providing superior performance and lower latency. The system creates a virtual service that appears as a single server to clients while distributing requests across multiple real servers behind the scenes.
LVS supports three primary forwarding methods:
- NAT (Network Address Translation): Modifies packet headers to redirect traffic
- DR (Direct Routing): Routes packets directly to real servers using MAC address manipulation
- TUN (IP Tunneling): Encapsulates packets in IP tunnels for forwarding
Each method offers different advantages depending on your network topology and performance requirements.
Prerequisites and Requirements
Before configuring LVS load balancing, ensure you have the following prerequisites:
System Requirements
- Linux distribution with kernel 2.4 or higher (recommended: CentOS 7/8, Ubuntu 18.04+, or RHEL 7/8)
- Root or sudo access on all servers
- Minimum 2GB RAM per server (4GB recommended for production)
- Network connectivity between all servers
- Static IP addresses for the director and real servers
Software Dependencies
```bash
For CentOS/RHEL systems
sudo yum install -y ipvsadm keepalived
For Ubuntu/Debian systems
sudo apt-get update
sudo apt-get install -y ipvsadm keepalived
```
Network Planning
Plan your network architecture with the following considerations:
- Virtual IP (VIP): The IP address clients will connect to
- Director IP: The IP address of the LVS director server
- Real Server IPs: IP addresses of backend servers
- Network segments: Ensure proper routing between components
LVS Architecture and Components
Core Components
LVS Director: The load balancer that receives client requests and distributes them to real servers. It maintains connection tables and applies scheduling algorithms.
Real Servers: Backend servers that process actual client requests. These servers run your applications or services.
IPVS (IP Virtual Server): The kernel module that implements the load balancing functionality within the Linux kernel.
Connection Flow
1. Client sends request to Virtual IP (VIP)
2. LVS Director receives the request
3. Director applies scheduling algorithm to select a real server
4. Request is forwarded using chosen method (NAT/DR/TUN)
5. Real server processes request and sends response
6. Response returns to client (path depends on forwarding method)
Installation and Initial Setup
Installing IPVS Tools
First, install the necessary tools on your LVS director server:
```bash
Check if IPVS module is loaded
lsmod | grep ip_vs
Load IPVS module if not present
sudo modprobe ip_vs
Install ipvsadm for management
sudo yum install -y ipvsadm # CentOS/RHEL
or
sudo apt-get install -y ipvsadm # Ubuntu/Debian
```
Enabling IP Forwarding
Enable IP forwarding on the director server:
```bash
Enable IP forwarding temporarily
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
Enable IP forwarding permanently
echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
```
Configuring Firewall Rules
Configure firewall rules to allow necessary traffic:
```bash
For firewalld (CentOS/RHEL)
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --permanent --add-port=80/tcp
sudo firewall-cmd --permanent --add-port=443/tcp
sudo firewall-cmd --reload
For iptables
sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT
sudo iptables -A FORWARD -j ACCEPT
```
Configuring LVS Director
Basic LVS Configuration
Create a basic LVS configuration using ipvsadm:
```bash
Clear existing rules
sudo ipvsadm -C
Add virtual service (HTTP)
sudo ipvsadm -A -t 192.168.1.100:80 -s rr
Add real servers
sudo ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.101:80 -m -w 1
sudo ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.102:80 -m -w 1
sudo ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.103:80 -m -w 1
Save configuration
sudo ipvsadm-save > /etc/sysconfig/ipvsadm
```
Configuration Parameters Explained
- `-A`: Add virtual service
- `-t`: TCP service (use `-u` for UDP)
- `192.168.1.100:80`: Virtual IP and port
- `-s rr`: Scheduling method (round-robin)
- `-a`: Add real server
- `-r`: Real server IP and port
- `-m`: NAT forwarding method (use `-g` for DR, `-i` for TUN)
- `-w 1`: Weight for load distribution
Advanced Director Configuration
Create a more sophisticated configuration script:
```bash
#!/bin/bash
LVS Director Configuration Script
VIP="192.168.1.100"
PORT="80"
REAL_SERVERS=("192.168.1.101" "192.168.1.102" "192.168.1.103")
Clear existing configuration
ipvsadm -C
Add virtual service with weighted round-robin scheduling
ipvsadm -A -t ${VIP}:${PORT} -s wrr
Add real servers with different weights
for i in "${!REAL_SERVERS[@]}"; do
weight=$((i + 1))
ipvsadm -a -t ${VIP}:${PORT} -r ${REAL_SERVERS[$i]}:${PORT} -m -w ${weight}
echo "Added real server: ${REAL_SERVERS[$i]} with weight: ${weight}"
done
Display configuration
ipvsadm -Ln
Save configuration
ipvsadm-save > /etc/sysconfig/ipvsadm
```
Setting Up Real Servers
Configuring Real Servers for NAT Mode
For NAT mode, configure real servers to use the LVS director as their default gateway:
```bash
On each real server
sudo route add default gw 192.168.1.10 # Director's internal IP
Make the route permanent
echo "GATEWAY=192.168.1.10" | sudo tee -a /etc/sysconfig/network
```
Configuring Real Servers for Direct Routing
For DR mode, configure the loopback interface on real servers:
```bash
#!/bin/bash
Script for real server DR configuration
VIP="192.168.1.100"
Configure loopback interface
sudo ifconfig lo:0 ${VIP} netmask 255.255.255.255 broadcast ${VIP} up
Add route
sudo route add -host ${VIP} dev lo:0
Suppress ARP responses for VIP
echo 1 | sudo tee /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 | sudo tee /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 | sudo tee /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 | sudo tee /proc/sys/net/ipv4/conf/all/arp_announce
Make ARP settings permanent
cat << EOF | sudo tee -a /etc/sysctl.conf
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
EOF
sudo sysctl -p
```
Installing and Configuring Web Services
Install and configure web services on real servers:
```bash
Install Apache/Nginx
sudo yum install -y httpd # CentOS/RHEL
or
sudo apt-get install -y apache2 # Ubuntu/Debian
Create unique content for testing
echo "
Real Server $(hostname -I)
" | sudo tee /var/www/html/index.html
Start and enable service
sudo systemctl start httpd
sudo systemctl enable httpd
```
Load Balancing Methods
Scheduling Algorithms
LVS supports various scheduling algorithms:
Round Robin (rr): Distributes requests equally among servers
```bash
sudo ipvsadm -A -t 192.168.1.100:80 -s rr
```
Weighted Round Robin (wrr): Considers server weights for distribution
```bash
sudo ipvsadm -A -t 192.168.1.100:80 -s wrr
```
Least Connections (lc): Routes to server with fewest active connections
```bash
sudo ipvsadm -A -t 192.168.1.100:80 -s lc
```
Weighted Least Connections (wlc): Combines connection count with weights
```bash
sudo ipvsadm -A -t 192.168.1.100:80 -s wlc
```
Forwarding Methods Comparison
| Method | Pros | Cons | Use Case |
|--------|------|------|----------|
| NAT | Simple setup, works with any server | Director bottleneck, limited scalability | Small to medium deployments |
| DR | High performance, scalable | Complex network setup | High-traffic environments |
| TUN | Works across subnets | Requires tunnel support | Geographically distributed servers |
Practical Configuration Examples
Example 1: Basic Web Load Balancer (NAT Mode)
Complete configuration for a simple web load balancer:
```bash
#!/bin/bash
Basic Web Load Balancer Configuration
Network Configuration
VIP="10.0.1.100"
DIP="10.0.1.10" # Director IP
RIP1="10.0.1.101" # Real Server 1
RIP2="10.0.1.102" # Real Server 2
RIP3="10.0.1.103" # Real Server 3
Clear existing configuration
ipvsadm -C
Configure virtual service for HTTP
ipvsadm -A -t ${VIP}:80 -s wrr
Add real servers
ipvsadm -a -t ${VIP}:80 -r ${RIP1}:80 -m -w 3
ipvsadm -a -t ${VIP}:80 -r ${RIP2}:80 -m -w 2
ipvsadm -a -t ${VIP}:80 -r ${RIP3}:80 -m -w 1
Configure virtual service for HTTPS
ipvsadm -A -t ${VIP}:443 -s wrr
Add real servers for HTTPS
ipvsadm -a -t ${VIP}:443 -r ${RIP1}:443 -m -w 3
ipvsadm -a -t ${VIP}:443 -r ${RIP2}:443 -m -w 2
ipvsadm -a -t ${VIP}:443 -r ${RIP3}:443 -m -w 1
Configure VIP on director
ip addr add ${VIP}/24 dev eth0
Enable connection tracking
ipvsadm --set 28800 120 300
Display configuration
ipvsadm -Ln --stats
echo "LVS configuration completed successfully!"
```
Example 2: High-Performance Setup (DR Mode)
Configuration for direct routing mode:
```bash
#!/bin/bash
High-Performance LVS Configuration (DR Mode)
VIP="192.168.10.100"
REAL_SERVERS=("192.168.10.101" "192.168.10.102" "192.168.10.103" "192.168.10.104")
Director Configuration
echo "Configuring LVS Director..."
Clear existing rules
ipvsadm -C
Add virtual service with locality-based scheduling
ipvsadm -A -t ${VIP}:80 -s lblc
ipvsadm -A -t ${VIP}:443 -s lblc
Add real servers for HTTP and HTTPS
for server in "${REAL_SERVERS[@]}"; do
ipvsadm -a -t ${VIP}:80 -r ${server}:80 -g -w 1
ipvsadm -a -t ${VIP}:443 -r ${server}:443 -g -w 1
echo "Added real server: ${server}"
done
Configure VIP on director (DR mode)
ip addr add ${VIP}/32 dev eth0:1
Optimize kernel parameters
sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv4.conf.all.send_redirects=0
sysctl -w net.ipv4.conf.default.send_redirects=0
sysctl -w net.ipv4.conf.eth0.send_redirects=0
echo "Director configuration completed!"
echo "Configure real servers with the DR setup script."
```
Example 3: Multi-Service Load Balancer
Configuration for multiple services:
```bash
#!/bin/bash
Multi-Service LVS Configuration
VIP="172.16.1.100"
Web servers
WEB_SERVERS=("172.16.1.101" "172.16.1.102")
Database servers
DB_SERVERS=("172.16.1.201" "172.16.1.202")
Mail servers
MAIL_SERVERS=("172.16.1.301" "172.16.1.302")
Clear configuration
ipvsadm -C
HTTP Service
ipvsadm -A -t ${VIP}:80 -s wrr
for server in "${WEB_SERVERS[@]}"; do
ipvsadm -a -t ${VIP}:80 -r ${server}:80 -m -w 1
done
HTTPS Service
ipvsadm -A -t ${VIP}:443 -s wrr
for server in "${WEB_SERVERS[@]}"; do
ipvsadm -a -t ${VIP}:443 -r ${server}:443 -m -w 1
done
MySQL Service
ipvsadm -A -t ${VIP}:3306 -s lc
for server in "${DB_SERVERS[@]}"; do
ipvsadm -a -t ${VIP}:3306 -r ${server}:3306 -m -w 1
done
SMTP Service
ipvsadm -A -t ${VIP}:25 -s rr
for server in "${MAIL_SERVERS[@]}"; do
ipvsadm -a -t ${VIP}:25 -r ${server}:25 -m -w 1
done
Save configuration
ipvsadm-save > /etc/sysconfig/ipvsadm
echo "Multi-service LVS configuration completed!"
ipvsadm -Ln
```
Monitoring and Management
Monitoring LVS Status
Create monitoring scripts to track LVS performance:
```bash
#!/bin/bash
LVS Monitoring Script
echo "=== LVS Status Report ==="
echo "Date: $(date)"
echo
Display virtual services
echo "Virtual Services:"
ipvsadm -Ln
echo
echo "Connection Statistics:"
ipvsadm -Ln --stats
echo
echo "Rate Statistics:"
ipvsadm -Ln --rate
echo
echo "Connection Table:"
ipvsadm -Lnc
Check real server health
echo
echo "Real Server Health Check:"
VIP="192.168.1.100"
REAL_SERVERS=("192.168.1.101" "192.168.1.102" "192.168.1.103")
for server in "${REAL_SERVERS[@]}"; do
if curl -s --connect-timeout 5 "http://${server}" > /dev/null; then
echo "✓ ${server}: Healthy"
else
echo "✗ ${server}: Unhealthy"
fi
done
```
Automated Health Checking
Implement automated health checking with keepalived:
```bash
/etc/keepalived/keepalived.conf
global_defs {
notification_email {
admin@example.com
}
notification_email_from lvs@example.com
smtp_server localhost
smtp_connect_timeout 30
router_id LVS_MASTER
}
virtual_server 192.168.1.100 80 {
delay_loop 6
lb_algo wrr
lb_kind NAT
persistence_timeout 50
protocol TCP
real_server 192.168.1.101 80 {
weight 3
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.1.102 80 {
weight 2
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
```
Performance Monitoring
Monitor system performance and LVS metrics:
```bash
#!/bin/bash
Performance Monitoring Script
LOG_FILE="/var/log/lvs-performance.log"
while true; do
timestamp=$(date '+%Y-%m-%d %H:%M:%S')
# Get connection count
connections=$(ipvsadm -Ln --stats | grep "TCP" | awk '{sum += $2} END {print sum}')
# Get system load
load=$(uptime | awk -F'load average:' '{print $2}' | awk '{print $1}' | sed 's/,//')
# Get memory usage
memory=$(free | grep Mem | awk '{printf "%.2f", $3/$2 * 100.0}')
# Log metrics
echo "${timestamp},${connections:-0},${load},${memory}" >> ${LOG_FILE}
sleep 60
done
```
Troubleshooting Common Issues
Connection Issues
Problem: Clients cannot connect to virtual IP
```bash
Check if VIP is configured
ip addr show | grep 192.168.1.100
Verify IPVS rules
ipvsadm -Ln
Check firewall rules
iptables -L -n | grep 80
Test connectivity to real servers
for server in 192.168.1.101 192.168.1.102; do
telnet $server 80
done
```
Problem: Uneven load distribution
```bash
Check current connections
ipvsadm -Ln --stats
Verify scheduling algorithm
ipvsadm -Ln | grep -E "TCP|UDP"
Check server weights
ipvsadm -Ln | grep "Route"
Reset connection table if needed
ipvsadm -C
Reconfigure services
```
Performance Issues
Problem: High latency or slow responses
```bash
Check system resources
top
iostat -x 1
Monitor network traffic
iftop -i eth0
Check connection table size
wc -l /proc/net/ip_vs_conn
Optimize connection tracking timeouts
ipvsadm --set 900 120 300
```
Problem: Real server overload
```bash
Check individual server performance
ssh user@192.168.1.101 'top; iostat'
Adjust server weights
ipvsadm -e -t 192.168.1.100:80 -r 192.168.1.101:80 -m -w 1
Add additional real servers
ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.104:80 -m -w 2
```
Network Configuration Issues
Problem: Direct routing not working
```bash
On real servers, check ARP settings
cat /proc/sys/net/ipv4/conf/all/arp_ignore
cat /proc/sys/net/ipv4/conf/all/arp_announce
Verify loopback configuration
ip addr show lo
Check routing table
route -n
Test ARP resolution
arping -I eth0 192.168.1.100
```
Debugging Tools and Commands
Essential debugging commands:
```bash
Monitor real-time connections
watch -n 1 'ipvsadm -Ln --stats'
Check kernel messages
dmesg | grep -i ipvs
Monitor network packets
tcpdump -i eth0 host 192.168.1.100
Check connection tracking
cat /proc/net/nf_conntrack | grep 192.168.1.100
Verify IPVS module
lsmod | grep ip_vs
modinfo ip_vs
```
Best Practices and Optimization
Performance Optimization
Kernel Parameter Tuning:
```bash
/etc/sysctl.conf optimizations
net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
Connection tracking optimizations
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 7200
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
Network buffer optimizations
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 65536 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
```
Connection Timeout Optimization:
```bash
Optimize IPVS connection timeouts
ipvsadm --set 900 120 300 # TCP timeout, TCP fin timeout, UDP timeout
```
Security Considerations
Firewall Configuration:
```bash
Restrict access to management interfaces
iptables -A INPUT -p tcp --dport 22 -s 192.168.1.0/24 -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j DROP
Rate limiting
iptables -A INPUT -p tcp --dport 80 -m limit --limit 25/minute --limit-burst 100 -j ACCEPT
```
Access Control:
```bash
Implement source IP restrictions
ipvsadm -A -t 192.168.1.100:80 -s wrr
ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.101:80 -m -w 1
Use connection limiting
echo 'net.ipv4.netfilter.ip_conntrack_max = 65536' >> /etc/sysctl.conf
```
High Availability Setup
Keepalived Integration:
```bash
Master configuration
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass secretpass
}
virtual_ipaddress {
192.168.1.100
}
}
```
Monitoring and Alerting
Log Rotation Configuration:
```bash
/etc/logrotate.d/lvs
/var/log/lvs-performance.log {
daily
rotate 30
compress
delaycompress
missingok
notifempty
create 644 root root
}
```
Automated Backup:
```bash
#!/bin/bash
LVS Configuration Backup Script
BACKUP_DIR="/backup/lvs"
DATE=$(date +%Y%m%d_%H%M%S)
mkdir -p ${BACKUP_DIR}
ipvsadm-save > ${BACKUP_DIR}/ipvsadm_${DATE}.conf
cp /etc/keepalived/keepalived.conf ${BACKUP_DIR}/keepalived_${DATE}.conf
Keep only last 30 days of backups
find ${BACKUP_DIR} -name "*.conf" -mtime +30 -delete
```
Advanced Configurations
Session Persistence
Configure session persistence for applications requiring sticky sessions:
```bash
Enable persistence with timeout (50 seconds)
ipvsadm -A -t 192.168.1.100:80 -s wrr -p 50
Configure persistence with netmask for subnet persistence
ipvsadm -A -t 192.168.1.100:80 -s wrr -p 300 -M 255.255.255.0
```
Firewall Marks
Use firewall marks for advanced traffic classification:
```bash
Mark packets with iptables
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p tcp --dport 443 -j MARK --set-mark 1
Create virtual service using firewall mark
ipvsadm -A -f 1 -s wrr
Add real servers
ipvsadm -a -f 1 -r 192.168.1.101:0 -m -w 1
ipvsadm -a -f 1 -r 192.168.1.102:0 -m -w 1
```
Geographic Load Balancing
Implement geographic load balancing with multiple LVS clusters:
```bash
#!/bin/bash
Geographic Load Balancing Setup
Local cluster (primary)
LOCAL_VIP="10.0.1.100"
LOCAL_SERVERS=("10.0.1.101" "10.0.1.102")
Remote cluster (backup)
REMOTE_VIP="10.0.2.100"
REMOTE_SERVERS=("10.0.2.101" "10.0.2.102")
Configure local cluster
ipvsadm -A -t ${LOCAL_VIP}:80 -s wrr
for server in "${LOCAL_SERVERS[@]}"; do
ipvsadm -a -t ${LOCAL_VIP}:80 -r ${server}:80 -m -w 2
done
Configure backup to remote cluster (lower weight)
ipvsadm -a -t ${LOCAL_VIP}:80 -r ${REMOTE_VIP}:80 -m -w 1
```
Conclusion
Linux Virtual Server (LVS) provides a robust, high-performance load balancing solution that can significantly improve the availability and scalability of your network services. This comprehensive guide has covered everything from basic installation and configuration to advanced optimization techniques and troubleshooting strategies.
Key takeaways from this guide include:
- Understanding the three forwarding methods (NAT, DR, TUN) and selecting the appropriate one for your environment
- Proper network configuration is crucial for successful LVS deployment
- Regular monitoring and health checking ensure optimal performance and availability
- Performance tuning through kernel parameters and connection timeout optimization
- Security considerations should be integrated from the beginning
- High availability can be achieved through keepalived integration
Next Steps
To further enhance your LVS deployment:
1. Implement comprehensive monitoring using tools like Nagios, Zabbix, or Prometheus
2. Set up automated failover with keepalived for high availability
3. Consider SSL termination strategies for HTTPS traffic
4. Explore integration with container orchestration platforms like Kubernetes
5. Plan for capacity scaling as your traffic grows
Additional Resources
- Official IPVS documentation: [Linux Virtual Server Project](http://www.linuxvirtualserver.org/)
- Keepalived documentation for high availability setups
- Performance tuning guides for high-traffic environments
- Security hardening best practices for load balancers
By following the practices and configurations outlined in this guide, you'll have a solid foundation for implementing and maintaining LVS load balancing in production environments. Remember to always test configurations in a development environment before deploying to production, and maintain regular backups of your configuration files.