[System] IP cluster and loadbalancing

IP clustering consists in creating a loadbalancing between many servers.
If we have many servers, it's possible to see them as a single server which balances resources and is crash proof.

It's possible to make clusters with any protocol as HTTP, MySQL and SMTP for example.
Creating clusters is simple and we are going to see how create a common part and their configurations.

Needed softwares
ipvsadm
ldirectord
A HTTP cluster configuration
A MySQL cluster configuration
A SMTP cluster configuration
Known issues

Needed softwares :

All softwares we are going to installed are famous in the opensource world and are available in a single package : UltraMonkey.
The only issue with the UltraMonkey package for Debian-like is that it exists only for the 32bits architecture and compile it is quiet hard.

However, it's possible to create clusters without UltraMonkey but installing them one by one.

First, the clustering tool : ipvsadm (from the Linux Virtual Server (LVS) project).
Ipvsadm allows to make a cluster with only one line.

Because ipvsadm allows "only" to manage clusters and servers into them, it's possible to use a software which will monitore these servers. This software is ldirectord.

ipvsadm

Once ipvsadm installed (via the apt-get install ipvsadm command), it's now possible to create clusters.
Cluster creation is made with a command line :

Cluster creation :

 ipvsadm -A -t 192.168.1.225:80 -s rr

Some explanations :

Cluster creation is made with the -A (Add) option
-t option is for TCP protocol. For an UDP protocol, use the -u option. It's possible to combine "redirections" with iptables using the -f option, if this, the iptable mark function will be used.
Then comes the cluster definition. Here, the cluster will work with the local IP 192.168.1.225 and with the 80 port.
The last option, -s, defined the loadbalancing algorithm between servers behing the cluster. Here, the chosen algorithm is Round-Robin (rr) but there are also Weighted Round-Robin (wrr), Least Connected (lc) and Weighted Least Connected (wlc) algorithms

Adding servers :

ipvsadm -a -t 192.168.1.225:80 -r 192.168.1.203:80 -g

Some explanations :

Adding a server is done with the -a (add) option
Then the service type (here TCP)
Then the cluster definition ( IP address and port)
Then the remote server to add ( IP address and port) with the -r option
And finally the connection type : -g means gate

Deleting a server from an cluster :

ipvsadm -d -t 192.168.1.225:80 -r 192.168.1.203:80

Some explanations :

-d is the flag to delete a server
-t is the service type defined during the server declaration in the cluster
192.168.1.225:80 is the given cluster
-r is the server to delete (means remove)

Clusters listing :

ipvsadm -ln

Some explanations :

The -l option allows to list all IP clusters installed and for each their servers
The -n option allows to not resolve the port / protocole (for example will display3306 instead of mysql)

Deleting a cluster :

ipvsadm -D -t 192.168.1.225:80

With these commands, it's now possible to manage an IP cluster.
But it's not enough to have a functionnal cluster. The server / cluster part is customizable but we must setting up servers which will be in the connection pool.

In order to add a server in a cluster, we must :

Setting up the kernel in the /etc/sysctl.conf file or adding an initscript

In the /etc/sysctl.conf file :

net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

In an initscript :

echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore 
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

Depending the chosen method, we have to execute both scripts or launch the sysctl -p command to load parameters.
Add for the servers in the cluster in the /etc/network/interfaces file :

# Interface LoadBalancee 
auto lo:225
iface lo:225 inet static         
         address 192.168.1.225         
         netmask 255.255.255.255         
         network 192.168.1.225         
         broadcast 192.168.1.225         
         up /sbin/route add -host 192.168.1.225 dev lo         
         down /sbin/route del -host 192.168.1.225

Then mount the new network interface : ifup lo:225

Because creating a cluster or adding / deleting server from this cluster isn't usefull with the command line and because if a server is down we have to remove it from the cluster, there is a very usefull tool : ldirectord.

ldirectord

ldirectord can be donloaded here or can be installed with a package ( apt-get install ldirectord ).
This tool provides facilities to create and manage a cluster. It works with a configuration file ( /etc/ha.d/ldirectord.cf ).

An internet manpage version is available here.

The ldirectord settings file is divided into 2 parts. The first on defines main parameters and the second on clusters themselves.

First part :

checktimeout=5 
checkinterval=10
autoreload=no
logfile="/var/log/ldirectord.log"
quiescent=no

Some explanations :

checktimeout : maximum waiting duration to declare a server is up
checkinterval : duration between 2 checks on a server in seconds
autoreload : configuration file hot update
logfile : ldirectord log file (add / delete servers in clusters)
quiescent : if yes, a server declared as down (e.g. doesn't responding on ping) won't be deleted in the cluster but will have a weight of 0 and doesn't accept new connections any more but will answer (if it can) on established connections.

There are a lot of others options but these one are really important...
All declared option can be override in a cluster definition.

Second part : clusters

virtual=192.168.1.225:80         
           real=192.168.1.203:80 gate 1
           real=192.168.1.204:80 gate 2
           service=http
           request="/index.html"
           receive="OK"
           scheduler=wrr
           protocol=tcp
           checktype=negotiate 
           fallback=192.168.1.205:80 gate

Some explanations :

virtual : the cluster defined by an IP address and its port / protocole
real : a server to add in a cluster defined by an IP address, its port / protocole and the communication mode. gate allows to connect directly a client with the server, masq allows to do masquerading between client and servers (to be activated with iptables). The last parameter is the weight given to the server
service is the service used by the cluster (for example http, mysql, smtp, ftp, ldap, etc...)
request : the request which must be done by the cluster to know if a server is up
receive : regular expression used by ldirectord on the request response to know if a server is up
scheduler : loadbalancer type (rr, wrr, lc, wlc, etc...)
protocol : used protocole (tcp, udp)
checktype : check type : negociate for a request usage and receive, ping for... ping, external for an external script usage (for example a customized bash script) combined with checkcommand
fallback : defines a server which will be add into the cluster if and only if the cluster is empty (all other servers are checked as down). This option is useful to use service webapp if there is a problem due to overload for example...

Following, examples of second part configuration for different kinds of services.

A HTTP cluster configuration :

Used for a SOLr searcher

virtual=192.168.200.225:80
           real=192.168.200.201:80 gate
           real=192.168.200.203:80 gate
           service=http
           request="/solr_blogs/admin/ping"
           receive="OK"
           scheduler=wrr
           protocol=tcp
           checktype=negotiate

A MySQL cluster configuration :

2 real servers in the cluster plus one additional in case first both are down. The first both servers are slaves of the third.
In this cluster, the /root/scripts/check_mysql.sh script checks if databases are available and replication delay.

The cluster :

virtual=192.168.1.152:3306
           real=192.168.1.102:3306 gate 100
           real=192.168.1.103:3306 gate 100
           fallback=192.168.1.101:3306 gate 1
           service=mysql
           scheduler=wrr
           checktype=external
           checkcommand="/root/scripts/check_mysql.sh"

The script :

#!/bin/bash

VIRTUAL_SERVER=$1
VIRTUAL_PORT=$2
REAL_SERVER=$3
REAL_PORT=$4

MAX_LATE=200

OK=0
KO=1

PID=$$
FILE=/tmp/mysql_${3}.${PID}

RETURN=$KO

RES=$(mysql -umonitoring -pmonitoring -h${REAL_SERVER} -N -e "SELECT 1;")


if [ "$RES" == "1" ];
then
    LATE=$( mysql -umonitoring -pmonitoring -h$3 -e "SHOW SLAVE STATUSG" | grep Seconds_Behind_Master | awk '{print $2}')

    if [ $LATE -lt $MAX_LATE ];
    then
            RETURN=$OK
        
           fi 
fi

if [ $RETURN -eq $OK ];
then
    echo "$REAL_SERVER OK" >> /tmp/cluster.sql
else
    echo "$REAL_SERVER NOK" >> /tmp/cluster.sql
fi

exit $RETURN

A second possibility without an external script but a simple SQL query :

virtual=192.168.1.152:3306
           real=192.168.1.102:3306 gate 100
           real=192.168.1.103:3306 gate 100
           fallback=192.168.1.101:3306 gate 1
           service=mysql
           login="monitoring"
           passwd="monitoring"
           scheduler=wrr
           checktype=negociate
           request="SELECT 1;"
           receive="1"
           database="pavnay"

A SMTP cluster configuration :

This cluster looks like the MySQL cluster but sending an email instead of querying databases.

The cluster :

virtual=192.168.1.153:25
           real=192.168.1.50:25 gate
           real=192.168.1.51:25 gate
           real=192.168.1.53:25 gate
           checkinterval=30
           scheduler=wrr
           protocol=tcp
           checktype=external
           checkcommand="/root/scripts/check_smtp.sh"

The script :

 #!/bin/bash

VIRTUAL_SERVER=$1
VIRTUAL_PORT=$2
REAL_SERVER=$3
REAL_PORT=25

PID=$$
FILE=/tmp/smtp_${REAL_SERVER}.${PID}

echo -n -e "HELO pavnayrnMAIL FROM: <webmaster@pavnay.fr>rnRCPT TO: <webmaster@pavnay.fr>rnDATArnFrom: ncrnTo: mernSubject: $3 - $(date)rn$(date)rn.rnQUITrn" | nc -t $REAL_SERVER $REAL_PORT>$FILE

COUNT=$(tail -3 $FILE | egrep -c -e "250 ok .+ qp" )

if [ "$COUNT" == "1" ];
then
    RETURN=0
else
    RETURN=1
fi

echo "CHECK MX - $3 - $(date) $COUNT / $RETURN" >> /tmp/cluster.mx

rm $FILE 
exit $RETURN

Known issues :

gate mode and ports :

We tried to put severals HTTP servers on an unique server, each one listening on its own port all in a cluster.
In the gate mode, it's not possible to setting differents ports to one IP address. Indeed, only the protocole defined on the cluster is considered (for example, the http protocole uses the 80 port and by the way, all servers in this cluster will be requested on the 80 port).
To bypass this, it's possible to use the masq mode but it's more difficult to setting up...

MySQL cluster and connections pool :

We are using Tomcat and the JDBC connections pool.
The problem is the connections TTL between the pool and the cluster.
To bypass this problem, just up the TTL on the cluster with the persistent option and a duration in seconds.