Last week I wrote an article about installing OpenIO, and now I'd like to go a step further by showing you how to add a load balancer in front of it. For the experiments in my home lab I use Odroid SBCs, but you can do the same with Raspberry Pis or x86 machines.
Why Load Balancers?
Before explaining how to configure the load balancer and make it work with OpenIO, I'd like to explain why you need a load balancer.
In a scale-out infrastructure, load balancers spread I/O operations across multiple nodes and help to improve infrastructure availability. OpenIO is a scale-out object store, and all data and I/O operations are distributed across the cluster backend. But, as is usually the case with this kind of architecture, the single front-end I/O operation is managed by a single node in the cluster. In practice, your application knows an IP address and sends requests just to that single address. Without a load balancer all traffic would be directed to a single node, creating a bottleneck.
The role of the load balancer is to publish a virtual interface (IP address) and redirect traffic to all the front-end nodes in the cluster. There are several ways to manage traffic redirection and the way you chose to do so depends on the characteristics of your infrastructure and applications. In the case of OpenIO, all the front-end services are stateless, meaning that the nodes do not retain any status or connection information about the communication between the client and the node itself. This simplifies things, and the load balancer can be configured with simple policies like round-robin.
For load balancing, we usually recommend HAProxy, a fairly common, feature-rich open-source solution.
How to Configure HAProxy for OpenIO
Configuring HAProxy is simple. For this tutorial I am assuming that the load balancer and cluster nodes are on the same network, and am configuring HAProxy for round robin, which is the most common configuration in production for OpenIO.
The load balancer has a physical interface with an IP address in the 192.169.1.0 network and its virtual public IP address will be 192.168.1.100, while the cluster nodes will be 192.168.1.1-3.
First, install the HAProxy package on the host you want to use as the load balancer. In Ubuntu it will be:
#apt-get install haproxy
Then add a virtual IP address to the server that will host the balancer. To do that you can just add these lines to /etc/network/interface:
auto eth0:1
iface eth0:1 inet static
address 192.168.1.100
netmask 255.255.255.0
Bring the virtual interface up with the command:
#Ifconfig eth0:1 192.168.1.100 up
Create a /etc/haproxy/haproxy.cfg file with the following lines:
global
log 127.0.0.1 local0 debug
chroot /var/lib/haproxy
nbproc 4
user haproxy
group haproxy
daemon
stats socket /run/haproxy/haproxy.sock mode 0660 level admin
stats timeout 30
# SSL
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
ssl-default-bind-options no-sslv3 no-tlsv10 no-tls-tickets
ssl-default-server-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
ssl-default-server-options no-sslv3 no-tlsv10 no-tls-tickets
tune.ssl.default-dh-param 2048
defaults
log global
unique-id-format %{+X}o %ci:%cp_%fi:%fp_%Ts_%rt:%pid
unique-id-header X-Unique-ID
mode http
option httplog
option dontlognull
option forwardfor
option log-separate-errors
timeout connect 5s
timeout client 60s
timeout server 60s
errorfile 400 /usr/share/haproxy/400.http
errorfile 403 /usr/share/haproxy/403.http
errorfile 408 /usr/share/haproxy/408.http
errorfile 500 /usr/share/haproxy/500.http
errorfile 502 /usr/share/haproxy/502.http
errorfile 503 /usr/share/haproxy/503.http
errorfile 504 /usr/share/haproxy/504.http
### OpenIO Swift proxy
# Public Access
frontend swift-public
bind 192.168.1.100:6007
log-format %ci:%cp [%t] %ft %b/%s w=%Tw/c=%Tc/r=%Tr/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %ID
capture request header Host len 60
capture request header X-Unique-ID len 46
capture request header X-Client-IP len 24
reqadd X-Forwarded-Proto: http
default_backend swift-backend
backend swift-backend
balance roundrobin
server swift-proxy1 192.168.1.1:6007 check inter 5s
server swift-proxy2 192.168.1.2:6007 check inter 5s
server swift-proxy3 192.168.1.3:6007 check inter 5s
### Conscience
frontend conscience
mode tcp
bind 192.168.1.100:6000 name conscience
log-format %ci:%cp [%t] %ft %b/%s w=%Tw/c=%Tc/t=%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq
default_backend conscience-backend
backend conscience-backend
mode tcp
option tcp-check
server conscience1 192.168.1.1:6000 check inter 5s
The second part of the file is the most important. This is the section where all the front-end and back-end services are mapped. Each sub-section starts with frontend <name-of-the-service> followed by backend <name-of-the-service>. As you can see, the configuration file is pretty simple.
Check your configuration with this command:
#haproxy -c -f /etc/haproxy/haproxy.cfg
And start the load balancer with this one:
#systemctl start haproxy
Point your application to the virtual IP (192.168.1.100 in our case) and you'll see the traffic pass through the load balancer and be equally distributed to the cluster nodes.
Key Takeaways
Most Linux sysadmins have experience with HAProxy and this short article doesn't tell them anything new. But I had to ask how to configure it when I started working more on OpenIO, and I thought that it could be useful for people who want to try our software and don’t yet have a lot of experience with scale-out infrastructures.
Once again, I have shown that OpenIO is within anyone's reach and can be deployed quickly for testing and - why not? - something more serious than that without issues or particular limitations.
