Why HAProxy?

Various articles published on the internet point out at the inability of AWS ELB (Elastic Load Balancer) to handle huge spike in connections. One of the methods to resolve this problem is by using HAProxy. HAProxy provides advanced options that ELB does not have leading to effective load balancing. Some of these options include:

– SSL Termination
– Real Time Logging
– Configurable Timeout Period (it’s set to 60 seconds in ELB)
– Uniform Resource Indicator (URI) based load balancing
– Integrating with other AWS services like Auto-Scaling, SNS, CloudWatch, Route53, etc.

Why the Failover Setup?

Since the HAProxy is to be installed on an Elastic Compute Cloud (EC2) instance, it acts as a Single Point of Failure i.e. if the EC2 instance stops, the entire application/database behind it stops running. To avoid such a situation its necessary to have a failover mechanism in place.

So, in this guide , we’re going to set up HAProxy failover using Keepalived for high availability in AWS.

 

 

HAProxy-Failover-Arch

Before moving ahead, lets explore a little about HAProxy and Keepalived.

HAProxy

It is a free and a reliable solution offering high availability, load balancing and proxying for TCP and HTTP-based applications.

Keepalived

It is a routing system that provides simple and robust facilities for load balancing and high-availability to Linux system and Linux based infrastructures. We’re going to use Keepalived’s VRRP unicast feature to detect the downtime of HAProxy.

Combine both, HAProxy and Keepalived, and you get an ultimate, low-cost, high availability solution!

Let’s get started!

Step 1 : Prepping up both the HAProxy servers for keepalived.

– Let’s assume that you already have 2 HAProxy servers in two different availability zones that are up and running

– Also assume that the two HAProxy servers we have are :

Server1 who’s internal ip is 1.1.1.1 and is in availability zone 1

Server2 who’s internal ip is 2.2.2.2 and is in availability zone 2

Both the servers must be placed in the public subnet & must be assigned public IP’s of their own.

– AWS CLI is installed and configured on both the HAProxy servers

– Also, open inbound connections for port 112 (VRRP) from the security group/s of both the servers

Step 2 : Setting up Keepalived on both the HAProxy servers

– Make sure you download the latest version of keepalived from the official site that supports VRRP unicast

– Perform the following actions on both the HAProxy servers.

# wget http://www.keepalived.org/software/keepalived-1.2.22.tar.gz
# tar zxvf keepalived-1.2.22.tar.gz
# cd keepalived-1.2.22/
# ./configure
# make && make install
# cp keepalived/etc/init.d/keepalived.init /etc/init.d/keepalived
# mkdir /etc/keepalived

– Now that we have keepalived installed on both the servers, we can start configuring keepalived for automatic failover.

Step 3 : Configuring keepalived for automatic failover.

We are going to use Server1 as the MASTER server and Server2 as the BACKUP server. So by default, the Elastic IP will be attached to Server1 until it goes out of service.

In case of a failure, Server2 will become the master and grab the Elastic IP for itself using master.sh script and continue serving requests. We need to set up our master and slave configs for keepalived slightly different as they both have different roles to play in.

The keepalived config is saved in /etc/keepalived/keepalived.conf

For Server1 (Master)

This is the content of the keepalived config on the Master HAProxy Server i.e. Server1 (1.1.1.1)

vrrp_script chk_haproxy

{

script “pidof haproxy”

interval 2

}

vrrp_instance VI_1

{

debug 2

interface eth0 # interface to monitor

state MASTER

virtual_router_id 1 # Assign one ID for this route

priority 101 # 101 on master, 100 on backup

unicast_src_ip 1.1.1.1 # My IP

unicast_peer

{

2.2.2.2 # peer IP

}

track_script

{

chk_haproxy

}

notify_master /etc/keepalived/master.sh

}

 

For Server2 (Backup)

This is the content of the keepalived config on the Master HAProxy Server i.e. Server2 (2.2.2.2)

vrrp_script chk_haproxy

{

script “pidof haproxy”

interval 2

}

vrrp_instance VI_1

{

debug 2

interface eth0 # interface to monitor

state BACKUP

virtual_router_id 1 # Assign one ID for this route

priority 100 # 101 on master, 100 on backup

unicast_src_ip 2.2.2.2 # My IP

unicast_peer

{

1.1.1.1 # peer IP

}

track_script

{

chk_haproxy

}

notify_master /etc/keepalived/master.sh

}

 

The vrrp_script is used to verify the other party is functioning correctly. We also have in our notify_master section a master.sh script

Step 4 : Adding the master.sh to both the servers

The master.sh script is a aws-cli based script that disassociates Elastic IP from the out-of-service server and assigns it to the new MASTER. It should be present on both the MASTER and the BACKUP server.

master.sh

#!/bin/bash

EIP=52.80.1.1 # Elastic IP to be associated

INSTANCE_ID=i-abc12cba # instance id on which the script is

aws ec2 disassociate-address –public-ip $EIP

aws ec2 associate-address –public-ip $EIP –instance-id $INSTANCE_ID

 

That’s it ! Now you can start keepalived service on both the servers.

# /etc/init.d/keepalived start

# chkconfig keepalived on

Step 5 : Testing the connectivity

Once you start keepalived on both the HAProxy servers , you can check the connectivity using the following command :

# tcpdump “ip proto 112”


FAQ’s


  • What if I dont see any output after executing tcpdump?

Check if port 112(VRRP) is open in the security groups of both the HAProxy servers

Check if the port 112(VRRP) is allowed in the Network ACL of the VPC

  • What if master.sh script is unable to assign IP to the master ?

Verify the instance ID of the servers

Check if aws-cli is installed on both the HAProxy Servers that is used to associate/disassociate the Elastic IP

14 Comments

  1. Manaswinee

    Very nice article to start with, hope we will get more elaboration soon

  2. Jeff

    the mouse speed up is very odd when you are not used to a page scrolling at that rate. I do have other comments, but i had to put that there first 🙂

  3. Jeff

    Thanks for the tutorial. I think something important to note is that both HAProxy servers will need either an outbound route from the public subnet or be created with their own public ips as the aws command will not complete without one or the other. If you have a recommended approach to this, i’d love to hear it, i’m a bit new to a full AWS cluster as i have set up. Its fully functional, but may not be the best practice.

    • DevOpsTech

      Hey Jeff, the thing you pointed out was right. We will have to keep both the HAProxy servers in a public subnet and have to assign public IP’s of their own.We have made changes to the above guide.Thanks for taking interest and pointing out the errors 🙂

  4. Imran

    These instructions only seem to be for EC2-Classic and not EC2-VPC.!

    Please can you kindly update. Thanks

    • DevOpsTech

      Hi Imran, the instructions provided above are for EC2-VPC.

      • mattimatti

        Unfortunately not true, the master.sh script consider parameters for EC2 classic.


        /usr/local/bin/aws ec2 disassociate-address --association-id $ASSOCIATION_ID

        /usr/local/bin/aws ec2 associate-address --instance-id $INSTANCEID --allow-reassociation --allocation-id $ALLOCATION_ID

  5. Frederic P

    Failover is automatic but we have a little downtime during disassociate/associate EIP?

    • DevOpsTech

      There wouldn’t be a downtime as such. Although it may result in current sessions being dropped.
      To overcome this, we can use some advanced session handling techniques that will make the application much more resilient.

  6. Alex B

    Thanks a lot for this great article!

    Question: how does this solution cope with increases in traffic? Any way to do (horizontal) (auto)scaling of the HAProxy instances? Or is vertical scaling the only option?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>


CAPTCHA Image
Reload Image