Various articles published on the internet point out at the inability of AWS ELB (Elastic Load Balancer) to handle huge spike in connections. One of the methods to resolve this problem is by using HAProxy. HAProxy provides advanced options that ELB does not have leading to effective load balancing. Some of these options include:
– SSL Termination
– Real Time Logging
– Configurable Timeout Period (it’s set to 60 seconds in ELB)
– Uniform Resource Indicator (URI) based load balancing
– Integrating with other AWS services like Auto-Scaling, SNS, CloudWatch, Route53, etc.
Why the Failover Setup?
Since the HAProxy is to be installed on an Elastic Compute Cloud (EC2) instance, it acts as a Single Point of Failure i.e. if the EC2 instance stops, the entire application/database behind it stops running. To avoid such a situation its necessary to have a failover mechanism in place.
So, in this guide , we’re going to set up HAProxy failover using Keepalived for high availability in AWS.
Before moving ahead, lets explore a little about HAProxy and Keepalived.
It is a free and a reliable solution offering high availability, load balancing and proxying for TCP and HTTP-based applications.
It is a routing system that provides simple and robust facilities for load balancing and high-availability to Linux system and Linux based infrastructures. We’re going to use Keepalived’s VRRP unicast feature to detect the downtime of HAProxy.
Combine both, HAProxy and Keepalived, and you get an ultimate, low-cost, high availability solution!
Let’s get started!
Step 1 : Prepping up both the HAProxy servers for keepalived.
– Let’s assume that you already have 2 HAProxy servers in two different availability zones that are up and running
– Also assume that the two HAProxy servers we have are :
Server1 who’s internal ip is 184.108.40.206 and is in availability zone 1
Server2 who’s internal ip is 220.127.116.11 and is in availability zone 2
Both the servers must be placed in the public subnet & must be assigned public IP’s of their own.
– AWS CLI is installed and configured on both the HAProxy servers
– Also, open inbound connections for port 112 (VRRP) from the security group/s of both the servers
Step 2 : Setting up Keepalived on both the HAProxy servers
– Make sure you download the latest version of keepalived from the official site that supports VRRP unicast
– Perform the following actions on both the HAProxy servers.
– Now that we have keepalived installed on both the servers, we can start configuring keepalived for automatic failover.
Step 3 : Configuring keepalived for automatic failover.
We are going to use Server1 as the MASTER server and Server2 as the BACKUP server. So by default, the Elastic IP will be attached to Server1 until it goes out of service.
In case of a failure, Server2 will become the master and grab the Elastic IP for itself using master.sh script and continue serving requests. We need to set up our master and slave configs for keepalived slightly different as they both have different roles to play in.
The keepalived config is saved in /etc/keepalived/keepalived.conf
For Server1 (Master)
This is the content of the keepalived config on the Master HAProxy Server i.e. Server1 (18.104.22.168)
For Server2 (Backup)
This is the content of the keepalived config on the Master HAProxy Server i.e. Server2 (22.214.171.124)
The vrrp_script is used to verify the other party is functioning correctly. We also have in our notify_master section a master.sh script
Step 4 : Adding the master.sh to both the servers
The master.sh script is a aws-cli based script that disassociates Elastic IP from the out-of-service server and assigns it to the new MASTER. It should be present on both the MASTER and the BACKUP server.
That’s it ! Now you can start keepalived service on both the servers.
Step 5 : Testing the connectivity
Once you start keepalived on both the HAProxy servers , you can check the connectivity using the following command :
- What if I dont see any output after executing tcpdump?
Check if port 112(VRRP) is open in the security groups of both the HAProxy servers
Check if the port 112(VRRP) is allowed in the Network ACL of the VPC
- What if master.sh script is unable to assign IP to the master ?
Verify the instance ID of the servers
Check if aws-cli is installed on both the HAProxy Servers that is used to associate/disassociate the Elastic IP