Configuring MongoDB Replica Sets With Keepalived
While working on developing PaaS.io, one of my primary objectives is to ensure that everything is configured to be truly Highly Available (HA). This means everything has a secondary, replicated, and has automated failover. This way it is nicely resilent and fault tolerant. Core infrastructure changes can be made without any affect to running applications, servers go down without issues. Sweet, huh?
When it came to setting up MongoDB, the obvious option was to go to using Replica Sets. They're like master-slave replication in RDBMSes but the master node is essentially floating. Client applications connect to the whole cluster, identify the master, and then start to speak to it.
By default, Cloud Foundry provisions a dedicated MongoDB instance for you and provides you the IP, port, and other credentials. This doesn't include replication, and focuses on single IP connections rather than replica sets.
However when using Replica Sets, the connection pattern in your code is
slightly different. You use a
ReplSetConnection instead of a
Connection, in ruby driver speak. So you need to know you're
connecting to a Replica Set if the host you're connecting to isn't the
master. If you only connect to the master though, you can use a
I was interested in maintainin full compatibility with Cloud Foundry's
out-of-the-box experience, so first looked at using
mongos in front of
replicas. It is normally used to front a sharded setup, but using it
without sharding is still a feature request.
So I set out looking at how to use a virtual IP have it track which node is the master. In this case, if one switches over, it will pick it up and move the virtual IP to follow. This would be mostly transparent to the application. The only handling in the app is the next query will fail, but quickly reconnect and its fine.
I'd already been using keepalived for HA within HAProxy, and one of the nice things it provides is the ability to define a script to run as a part of its local health checks. Using this, can have a script that just asks the local system "you the master?" and returns appropriately.
Below is the script:
This won't return any output, but will exit with 0 if it is the master or 1 if it isn't.
Its requirements are simple... Ruby, RubyGems,
mongo gem, and
bson_ext gem too.
Then within our
keepalived.conf file, it looks like this:
With this in place, if a server goes down, it'll switch. If
dies, it will switch over (since Mongo itself will recognize that), and
if the current primary is switched, the
return an exit code of 1, causing it to switch over.
Another reason this might be useful in some cases is because some clients still don't support replica sets. With this method, they don't need to.
In addition to these steps, you will need to perform the normal steps to setup keepalived. These would include:
mongodis bound to
0.0.0.0so it'll accept any incoming connection.
net.ipv4.ip_nonlocal_bind=1in sysctl so you can use virtual IPs.
Try this out and let me know your experiences. If you're interested in this kind of seamless infrastructure automation, check out PaaS.io or drop me a line.