In Valid Logic

While working on developing PaaS.io, one of my primary objectives is to ensure that everything is configured to be truly Highly Available (HA). This means everything has a secondary, replicated, and has automated failover. This way it is nicely resilent and fault tolerant. Core infrastructure changes can be made without any affect to running applications, servers go down without issues. Sweet, huh?

When it came to setting up MongoDB, the obvious option was to go to using Replica Sets. They're like master-slave replication in RDBMSes but the master node is essentially floating. Client applications connect to the whole cluster, identify the master, and then start to speak to it.

By default, Cloud Foundry provisions a dedicated MongoDB instance for you and provides you the IP, port, and other credentials. This doesn't include replication, and focuses on single IP connections rather than replica sets.

However when using Replica Sets, the connection pattern in your code is slightly different. You use a ReplSetConnection instead of a normal Connection, in ruby driver speak. So you need to know you're connecting to a Replica Set if the host you're connecting to isn't the master. If you only connect to the master though, you can use a traditional connection.

I was interested in maintainin full compatibility with Cloud Foundry's out-of-the-box experience, so first looked at using mongos in front of replicas. It is normally used to front a sharded setup, but using it without sharding is still a feature request.

So I set out looking at how to use a virtual IP have it track which node is the master. In this case, if one switches over, it will pick it up and move the virtual IP to follow. This would be mostly transparent to the application. The only handling in the app is the next query will fail, but quickly reconnect and its fine.

I'd already been using keepalived for HA within HAProxy, and one of the nice things it provides is the ability to define a script to run as a part of its local health checks. Using this, can have a script that just asks the local system "you the master?" and returns appropriately.

Below is the script:

#!/usr/bin/env ruby

require 'optparse'

options = { :hostname => '127.0.0.1',
            :port     => 27017,
            :username => nil,
            :password => nil }

optparser = OptionParser.new do |opt|
  opt.banner = "Usage: #{$0} [options]"
  opt.on('-H', '--hostname HOST', 'Hostname') { |o| options[:hostname] = o if o }
  opt.on('-P', '--port PORT', 'Hostname') { |o| options[:port] = o.to_i if o }
  opt.on('-u', '--username USER', 'Username') { |o| options[:username] = o if o }
  opt.on('-p', '--password PASSWORD', 'Password') {|o| options[:password] = o if o }
end

begin
  optparser.parse!
rescue => e
  $stderr.print e
  $stderr.print optparser
  exit 1
end

require 'rubygems'
require 'mongo'

conn = Mongo::Connection.new(options[:hostname], options[:port])
conn.db('admin').authenticate(options[:username], options[:password])
master_status = conn.db('admin').command({'isMaster'=>1})

if master_status['ismaster']
  exit 0
else
  exit 1
end

This won't return any output, but will exit with 0 if it is the master or 1 if it isn't.

Its requirements are simple... Ruby, RubyGems, mongo gem, and recommend the bson_ext gem too.

Then within our keepalived.conf file, it looks like this:

global_defs {
   notification_email {
     me@example.com
   }
   notification_email_from system@example.com
   smtp_server 192.168.1.1
   smtp_connect_timeout 30
}

# Define the script used to check if mongod is running
vrrp_script chk_mongod {
    script "killall -0 mongod"
    interval 2 # every two seconds
    weight 2
}

# Define the script to see if the local node is the primary
vrrp_script chk_mongo_primary {
    script "/usr/local/mongodb/bin/mongo_check_primary -u admin -p password"
    interval 2 # every two seconds
    weight 2
}

# Configuation for the virtual interface
vrrp_instance VI_1 {
    interface bond0
    state node MASTER        # SLAVE on the other nodes
    priority 101             # 100 on other nodes
    virtual_router_id 55

    authentication {
        auth_type AH
        auth_pass secret     # Set this to some secret phrase
    }

    # The virtual ip address shared between the two nodes
    virtual_ipaddress {
        192.168.1.222
    }

    # Use the script above to check if we should fail over
    track_script {
        chk_mongod
        chk_mongo_primary
    }
}

With this in place, if a server goes down, it'll switch. If mongod dies, it will switch over (since Mongo itself will recognize that), and if the current primary is switched, the mongo_check_primary will return an exit code of 1, causing it to switch over.

Another reason this might be useful in some cases is because some clients still don't support replica sets. With this method, they don't need to.

In addition to these steps, you will need to perform the normal steps to setup keepalived. These would include:

• Ensure mongod is bound to 0.0.0.0 so it'll accept any incoming connection.
• Set net.ipv4.ip_nonlocal_bind=1 in sysctl so you can use virtual IPs.

Try this out and let me know your experiences. If you're interested in this kind of seamless infrastructure automation, check out PaaS.io or drop me a line.