In Valid Logic

Deploying Ubuntu 12.04 on XenServer Made Easy

Tue May 01 00:00:00 -0700 2012

To follow up on my previous post about disk errors with Ubuntu 12.04 on XenServer, I wanted to cover the process I've put in place for provisioning Ubuntu VMs.

With PaaS.io, I have a mixture between systems deployed on bare metal and virtualized. With the virtualized systems, I set out to make the provisioning as easy as if I was using an IaaS provider, while still giving me fine control over sizing and placement. Some of the decisions there are enough for another post.

Ubuntu 12.04 is the new hotness and I'd been anxiously awaiting it, with plans to progressively roll it out throughout PaaS.io.

With XenServer, it provides an easy template for provisioning Ubuntu Lucid 10.04 VMs and even for some newer releases. However, the way XenSever provisions Ubuntu is to essentially netboot it and install it from a remote source. Because of that, we can't simply use one of those templates buts install a newer release.

Fortunately, the template itself is very simple, and which release it installs is just a configuration parameter.

To create a new Ubuntu 12.04 template, simply log into the XenServer console and runt he following commands. It will clone the Lucid template and then change the parameter for the release to install from lucid to precise.

$ TEMPLATE_UUID=`xe template-list \
        name-label="Ubuntu Lucid Lynx 10.04 (64-bit)" params=uuid --minimal`
$ NEW_TEMPLATE_UUID=`xe vm-clone uuid=$TEMPLATE_UUID \
        new-name-label="Ubuntu Precise (64-bit)"`
$ xe template-param-set other-config:default_template=true \
        other-config:debian-release=precise uuid=$NEW_TEMPLATE_UUID

Now you will find a "Ubuntu Precise (64-bit)" option in the template list of XenCenter.

Now, we can actually provision a new box. Next, one interesting discovery was that you can pass in a kickstart script in the boot parameters options for the new VM.

Kickstart allows you to do a scripted installation, automated pretty much every aspect of the system. Instead of picking a bunch of options, it allows for an easy, repeatable process that basically leaves the machine completely ready to go.

To use a kickstart script, make a script available over HTTP somewhere on your LAN or on the public internet. By the time it is grabbed, the machine will have an IP, DNS, and all. Then, simply add ks=http://url/to/your/kickstart to the beginning part of "advanced OS boot parameters" option when selecting the installation media.

Below is a cleaned version of the kickstart script I used on my Ubuntu VMs. The main things of note:

Sets up my partition table
Configures base system with ubuntu-minimal as well as some common packages like openssh-server, curl, wget, and screen.
Disables the creation of the ubuntu user (I create a normal every day user through chef)
Configures the fstab with barrier=0 as mentioned before
Disables /bin/sh from pointing to /bin/dash (personal preference)
Updates apt sources
Installs xenstore-utils
Downloads some auto-configuration scripts
Installs XenTools
Installs the Ubuntu virtual kernel and removes the generic one
Cleans up apt caches
Shuts down the VM.

lang en_US
langsupport en_US
keyboard us
timezone America/Los_Angeles
text
install
skipx
halt
url --url http://us.archive.ubuntu.com/ubuntu

rootpw pa$$word   # you should replace, and use --iscrypted
auth --useshadow --enablemd5
user --disabled

bootloader --location=mbr
zerombr yes
clearpart --all --initlabel
part /boot --fstype=ext2 --size=64
part swap --size=1024
part / --fstype=ext4 --size=1 --grow

network --device=eth0 --bootproto=static --ip=10.0.0.50 --netmask=255.255.255.0 \
        --nameserver=10.0.0.1 --gateway=10.0.0.1
firewall --disabled

%packages
ubuntu-minimal
openssh-server
screen
curl
wget

%post

# update fstab for the root partition
perl -pi -e 's/(errors=remount-ro)/noatime,nodiratime,$1,barrier=0/' /etc/fstab

# point sh to bash instead of dash
rm /bin/sh
ln -s /bin/bash /bin/sh

# add normal apt source list
(
cat <<'EOP'
deb http://us.archive.ubuntu.com/ubuntu/ precise main restricted universe
deb http://us.archive.ubuntu.com/ubuntu/ precise-security main restricted universe
deb http://us.archive.ubuntu.com/ubuntu/ precise-updates main restricted universe
EOP
) > /etc/apt/sources.list
apt-get update
apt-get upgrade -y

# install some additional packages
apt-get install -y xenstore-utils

# set up xenserver automation scripts
AUTOMATER_REPO=https://raw.github.com/krobertson/xenserver-automater
curl $AUTOMATER_REPO/master/usr/sbin/xe-set-hostname > /usr/sbin/xe-set-hostname
curl $AUTOMATER_REPO/master/usr/sbin/xe-set-network > /usr/sbin/xe-set-network
curl $AUTOMATER_REPO/master/usr/sbin/generate-sshd-keys > /usr/sbin/generate-sshd-keys
curl $AUTOMATER_REPO/master/etc/init/xe-automator.conf > /etc/init/xe-automator.conf
chmod a+x /usr/sbin/xe-set-hostname
chmod a+x /usr/sbin/xe-set-network
chmod a+x /usr/sbin/generate-sshd-keys

# setup locales
locale-gen en_US.UTF-8
update-locale LANG="en_US.UTF-8"
echo 'LANG=en_US.UTF-8' >> /etc/environment
echo 'LC_ALL=en_US.UTF-8' >> /etc/environment

# install xe tools
cd /tmp
wget http://some/url/to/xe-guest-utilities_6.0.0-743_amd64.deb
dpkg -i xe-guest-utilities_6.0.0-743_amd64.deb

# install paravirt kernel image
apt-get install -f -y linux-virtual
dpkg -l | grep generic | grep linux | awk '{print $2}' | xargs apt-get remove -y

# clean up some stuff
rm -f /etc/ssh/ssh_host_*
rm -f /var/cache/apt/archives/*.deb
rm -f /var/cache/apt/*cache.bin
rm -f /var/lib/apt/lists/*_Packages

It is important that it shuts down at the end. My goal was to have it be all inclusive, and that means setting up the virtual kernel. However, it can't successfully reboot, because we need to update some PV boot options before it is ready to go.

Also, some packages must be installed in the post steps. The default install sources for Ubuntu don't always have all packages available, and I found it best to do the kernel last.

When you first bring up the new VM in XenServer, you may need to enter in a few details if you don't have DHCP running. It will self configure by default, but I normally opt to manually configure it to ensure the template gets a certain IP just to avoid any future possible collision.

After the bootstrapping is done and the VM is then off, log into console on the XenServer host itself and run the following snippet:

$ VMNAME=precise-20120501
$ UUID=`xe vm-list name-label="$VMNAME" params=uuid --minimal`
$ EDITOR=cat xe-edit-bootloader -n "$VMNAME" -p 1 \
        -f /grub/grub.cfg > /tmp/$VMNAME-grub
$ KERNEL=`grep vmlinuz /tmp/$VMNAME-grub | grep virtual |
        grep -v recovery | awk '{print $2}'`
$ ROOT=`grep vmlinuz /tmp/$VMNAME-grub | grep virtual |
        grep -v recovery | awk '{print $3}'`
$ RAMDISK=`grep initrd /tmp/$VMNAME-grub | head -1 | awk '{print $2}'`
$ xe vm-param-set uuid=$UUID PV-bootloader-args="--kernel=$KERNEL --ramdisk=$RAMDISK"
$ xe vm-param-set uuid=$UUID PV-args="$ROOT ro quiet console=hvc0"

Before you run it, of course set the VMNAME to the name of your VM. You might also want to check some of the values as you go (echo $UUID). You can use the xe-edit-bootloader command to view the grub configuration within the VM, but need to set the PV settings outside the guest. By faking the EDITOR to cat, you can export the grub file to a local file, then use some grep+awk to get the necessary pieces and finally set the PV settings correctly.

At this point, the machine is ready to be booted, converted into a template, or simply cloned. I personally like to leave my templates as never-been-used.

One of the main benefits of an IaaS setup like OpenStack or CloudStack is the self orientating of the VMs. Normally with a template, it keeps the template's setting for its hostname and network configuration. However, I found some example scripts on Github for passing network information into a VM through the VM's "xenstore" options. That way on boot, the VM can read in the settings it needs and update itself. I did some work to update the scripts to work with Precise, to more thoroughly update the hostname and DNS, as well as to regenerate the ssh host keys (since the kickstart did delete them).

As an example, this would be how to clone the template to a VM, set the params, and boot it:

$ UUID=`xe vm-install template=precise-20120501 new-name-label=builder`
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/ip=10.0.0.11
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/gw=10.0.0.1
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/nm=255.255.255.0
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/ns='10.0.0.1 10.0.0.2'
$ xe vm-start uuid=$UUID

That is cool and all, but I don't want to log into the XenServer console each time I want to setup a VM. Luckily, found a plugin for Chef's knife utility that adds XenServer provisioning. I forked it and added setting xenstore network parameters to it.

Now, provisioning a new VM, configuring it, and bootstrapping chef on it is just a matter of one call:

$ knife xenserver vm create --vm-template precise-20120501 \
                            -x root --keep-template-networks \
                            -r "role[foo]" -E production \
                            --vm-ip 10.0.0.12 --vm-name foobar

The command may be a little long, but it is all encapsulated in a single command. I simply have an internal README of sorts with the command pre-prepared for various roles.

If you have any questions, please leave a comment and ask. I'd be glad to help and always looking to improve my process as well.

Fixing random disk errors with Ubuntu 12.04 Precise on XenServer

Sat Apr 28 00:00:00 -0700 2012

Ubuntu 12.04 LTS Precise is hot off the press right now. Over the past few days, have been working on building new base images for PaaS.io, but was running into random issues where the root parition would encounter an error and freeze up. It normally happened just after it would finish booting and about 50% of the time.

A few times it happened after I already logged in, so was able to do basic read only operations. In dmesg, was able to see the following:

[    6.748868] blkfront: barrier: empty write xvda op failed
[    6.748876] blkfront: xvda: barrier or flush: disabled
[    6.748890] end_request: I/O error, dev xvda, sector 6584768
[    6.748908] end_request: I/O error, dev xvda, sector 6584768
[    6.748943] Aborting journal on device xvda6-8.
[    6.767022] EXT4-fs error (device xvda6): ext4_journal_start_sb:327: Detected aborted journal
[    6.767046] EXT4-fs (xvda6): Remounting filesystem read-only

Or if you weren't able to log in first, you might see this in the XenCenter console:

After some Googling, was able to track a similar error down by someone else. The fix was to update the mount options for the root partition. Mine are now:

noatime,nodiratime,errors=remount-ro,barrier=0

The key is the barrier=0. From some documentation, it is an option to help increase the integrity of writes by ensuring everything is flushed to disk be committing to the journal. However sometimes in a virtualized environment that is difficult to guarantee. In my case, have disk->RAID->dom0->LVM->domU.

Figure many other people will be diving into Precise this weekend, potentially running into this issue like me.

Soon, will post some additional details about how to easily get a nice Precise template in XenServer 6. I've been getting my setup nicely tuned using a kickstart script for the base system and leveraging xenstore data to dynamically setup the IP and hostname on boot.

Configuring MongoDB Replica Sets With Keepalived

Thu Feb 23 00:00:00 -0800 2012

While working on developing PaaS.io, one of my primary objectives is to ensure that everything is configured to be truly Highly Available (HA). This means everything has a secondary, replicated, and has automated failover. This way it is nicely resilent and fault tolerant. Core infrastructure changes can be made without any affect to running applications, servers go down without issues. Sweet, huh?

When it came to setting up MongoDB, the obvious option was to go to using Replica Sets. They're like master-slave replication in RDBMSes but the master node is essentially floating. Client applications connect to the whole cluster, identify the master, and then start to speak to it.

By default, Cloud Foundry provisions a dedicated MongoDB instance for you and provides you the IP, port, and other credentials. This doesn't include replication, and focuses on single IP connections rather than replica sets.

However when using Replica Sets, the connection pattern in your code is slightly different. You use a ReplSetConnection instead of a normal Connection, in ruby driver speak. So you need to know you're connecting to a Replica Set if the host you're connecting to isn't the master. If you only connect to the master though, you can use a traditional connection.

I was interested in maintainin full compatibility with Cloud Foundry's out-of-the-box experience, so first looked at using mongos in front of replicas. It is normally used to front a sharded setup, but using it without sharding is still a feature request.

So I set out looking at how to use a virtual IP have it track which node is the master. In this case, if one switches over, it will pick it up and move the virtual IP to follow. This would be mostly transparent to the application. The only handling in the app is the next query will fail, but quickly reconnect and its fine.

I'd already been using keepalived for HA within HAProxy, and one of the nice things it provides is the ability to define a script to run as a part of its local health checks. Using this, can have a script that just asks the local system "you the master?" and returns appropriately.

Below is the script:

#!/usr/bin/env ruby

require 'optparse'

options = { :hostname => '127.0.0.1',
            :port     => 27017,
            :username => nil,
            :password => nil }

optparser = OptionParser.new do |opt|
  opt.banner = "Usage: #{$0} [options]"
  opt.on('-H', '--hostname HOST', 'Hostname') { |o| options[:hostname] = o if o }
  opt.on('-P', '--port PORT', 'Hostname') { |o| options[:port] = o.to_i if o }
  opt.on('-u', '--username USER', 'Username') { |o| options[:username] = o if o }
  opt.on('-p', '--password PASSWORD', 'Password') {|o| options[:password] = o if o }
end

begin
  optparser.parse!
rescue => e
  $stderr.print e
  $stderr.print optparser
  exit 1
end

require 'rubygems'
require 'mongo'

conn = Mongo::Connection.new(options[:hostname], options[:port])
conn.db('admin').authenticate(options[:username], options[:password])
master_status = conn.db('admin').command({'isMaster'=>1})

if master_status['ismaster']
  exit 0
else
  exit 1
end

This won't return any output, but will exit with 0 if it is the master or 1 if it isn't.

Its requirements are simple... Ruby, RubyGems, mongo gem, and recommend the bson_ext gem too.

Then within our keepalived.conf file, it looks like this:

global_defs {
   notification_email {
     me@example.com
   }
   notification_email_from system@example.com
   smtp_server 192.168.1.1
   smtp_connect_timeout 30
}

# Define the script used to check if mongod is running
vrrp_script chk_mongod {
    script "killall -0 mongod"
    interval 2 # every two seconds
    weight 2
}

# Define the script to see if the local node is the primary
vrrp_script chk_mongo_primary {
    script "/usr/local/mongodb/bin/mongo_check_primary -u admin -p password"
    interval 2 # every two seconds
    weight 2
}

# Configuation for the virtual interface
vrrp_instance VI_1 {
    interface bond0
    state node MASTER        # SLAVE on the other nodes
    priority 101             # 100 on other nodes
    virtual_router_id 55

    authentication {
        auth_type AH
        auth_pass secret     # Set this to some secret phrase
    }

    # The virtual ip address shared between the two nodes
    virtual_ipaddress {
        192.168.1.222
    }

    # Use the script above to check if we should fail over
    track_script {
        chk_mongod
        chk_mongo_primary
    }
}

With this in place, if a server goes down, it'll switch. If mongod dies, it will switch over (since Mongo itself will recognize that), and if the current primary is switched, the mongo_check_primary will return an exit code of 1, causing it to switch over.

Another reason this might be useful in some cases is because some clients still don't support replica sets. With this method, they don't need to.

In addition to these steps, you will need to perform the normal steps to setup keepalived. These would include:

Ensure mongod is bound to 0.0.0.0 so it'll accept any incoming connection.
Set net.ipv4.ip_nonlocal_bind=1 in sysctl so you can use virtual IPs.

Try this out and let me know your experiences. If you're interested in this kind of seamless infrastructure automation, check out PaaS.io or drop me a line.

Jekyll on PaaS.io with Cloud Foundry

Fri Jan 06 00:00:00 -0800 2012

Recently I moved my blog over to the service I am currently working on building, PaaS.io.

Running Jekyll on PaaS.io isn't all that different from running it on other services, though I had a few other goals in mind. The way I wanted it set up was:

Stop using rack-jekyll. Its a nice gem, however it is locked to an older version of Jekyll. And the current gem is on an even more outdated one. Currently, Cloud Foundry doesn't support pulling bundler sources from git too.
Have a public folder for static content like CSS and images.
Have the _site folder for generated content
Don't have it copy the Gemfile and the config.ru into the _site folder (annoys me)
Redirect www.invalidlogic.com to invalidlogic.com
Low foot print

First, the Gemfile:

source :rubygems

gem 'rack-contrib', :require => 'rack/contrib/try_static'
gem 'rack-redirect'
gem 'thin'

group :development do
  gem 'jekyll'
  gem 'RedCloth'
  gem 'rdiscount'
end

The main gems being used are thin, rack-contrib (for TryStatic, note on that later), and rack-redirect (for www redirection). I also include some of the gems I use for Jekyll in the development group. That way they are available locally but not loaded when I deploy (lower footprint... and yes, it is minor).

Now, the config.ru:

require 'rubygems'
require 'bundler'
Bundler.require

use Rack::EY::Solo::DomainRedirect

use Rack::TryStatic,
    :root => "_site",
    :urls => %w[/],
    :try => ['.html', 'index.html', '/index.html']

use Rack::Static,
    :root => "public",
    :urls => %w[/]

run lambda { [404, {'Content-Type' => 'text/html'}, ['Not Found']]}

There are 4 rack components here. First, Rack::EY::Solo::DomainRedirect is the rack-redirect gem and handles the www redirection. Next, is Rack::TryStatic. It is used to access files from the _site generated content directory. It gives a couple different :try values for different ways to find the intended file. Then is the Rack::Static which gets static content from the public directory. No need to try different combinations. And last is a generic lambda that will return 404.

Next, want to avoid duplication. With things as they are, when I run jekyll it will copy the public and other items into the _site folder duplicating it. To resolve that, in our _config.yml, can add an exclude line:

exclude: [ 'public', 'Gemfile', 'Gemfile.lock', 'config.ru' ]

And with that, we are set! All of our goals are met. Commit and push to deploy! Currently I have Cloud Foundry set up with a Rack framework defined (will be sending a pull request with it soon) and also have my blog set to use Ruby 1.9.3 as well.

Soon I'll be providing some more details on PaaS.io, so stay tuned and click over to it and sign up to get access to the beta.

iPhone vs Android is the new Mac vs PC

Tue Oct 18 00:00:00 -0700 2011

After spending 18 months with Android, I am now back to an iPhone and likely here to stay. While Android isn't necessarily bad, it more boils down to iPhone simply being better. After a while, I started looking at iPhone vs Android as largely a repeat of the Mac vs PC comparisons.

Mac vs PC

Just look at it. Apple is doing what they've always done. They control the hardware and the software. They have a solid, unified experience. You can pick up any iPhone, old or new, and still be at home.

Android is repeating the history of PCs. Google makes the OS, as Microsoft did with Windows. Then OEMs offer a wide variety of different hardware, with their own shitty customizations layered on top of the OS. Even worse, you have the carriers layering on their customizations and restrictions.

You can go from one Android phone to another and have it be completely foreign. Even baseline apps can have different names and a completely different look and feel, such between the "Email" vs "Mail" of vanilla Android and HTC Android.

Shelf Life

With Android, the phones have a very short shelf life. I bought a Thunderbolt from Verizon back in April, just 6 months ago. About a month after I bought it, it was no longer the hot model they were pimping. In fact, there have been 2-3 phones to come out since then that became "king of the mountain."

With iPhone, they release a new phone about once a year, and that one stays the current phone. Older phones are still pretty well supported. The iPhone 3GS is over 2 years old, still got updated to iOS 5, and likely will until iOS 6. My original Motorola Droid that is nearing 2 years old is pretty much forgotten already.

Updates with iPhone? Available right away to everyone. AT&T or Verizon, you get the update when Apple makes it generally available.

Updates with Android? Have fun. Google has to release it, your OEM has to customize it, then the carrier gets to tweak it, and decide when to finally roll it out. Google released Gingerbread back in December 2010, nearly 10 months ago. Verizon just started rolling out the update last month, then pulled it. Even then, they decide when you can upgrade with a phased roll out. And since it was pulled, looks like they seem to skip over adequate testing.

Most Android users who want recent releases end up rooting their phones and use unofficial ROMs put together by an informal group of people. Have an issue with your phone? Limited options.

Marketing

I definitely agreed with my friend Joe in his post about the Droid Bionic. Android phones are being pitched like PCs. They give a bunch of technical specs that are meaningless to consumers. It echos the "Golden Circle" idea by Simon Sinek. Apple is still doing what they do best. Verizon sells like PC manufacturers. Lacking inspiration.

False Sense of Market Share

You've probably seen the headlines of "Android sales outpacing iPhone" or "Android market share to surpass iPhone".

Overall the numbers are comparing apples to oranges. They are comparing the sales figure of a single phone to a whole group of phones. There are many different Android phones, and individually, iPhone is spanking them in sales figures. No single Android phone has a chance of elipsing the iPhone or gaining any meaningful market share, especially with their overall short shelf life. I'd be very interested to see sales figured of individual phones and see how long they really last. All the manufacturers are still struggling to get a sliver of what the iPhone is capable of.

The numbers may hold some weight for developers because it represents the size of your audience. But at the same time, Android is many phones vs iPhones entire existence is now just 5 phones. Less "your app doesn't work on my Verizon Droid Mumbojumbo" and you don't have a Droid Mumbojumbo to test on.

Falseness of Open

Many people tout Android as being open, however the actions of Google with Honeycomb's source code seem to be heading in the direction of more closed.

While Google said part of their goal was to try and unify the platform more, as Honeycomb was intended for tablets and not handsets, the ability to control a platform is difficult while also keeping it "open." I think Google is right in closing it, since in order to further the platform, they will need to have some control in order to maintain a consistent direction.

However, the "openness" usually just comes from developers. What do most of then define the "openness" as? Being able to write apps and put them on their phone without paying $99. They tout the source being open, but the truth is that isn't what they really care about. Very few Android developers likely dive into the OS code, they just want to install their own apps for free.

Personally, if I prefer one platform over the other, a $99 fee to build apps isn't going to a stopper for me. But I also don't mind paying for the tools I use in my craft if they are worth it.

Working

Most average people care more about how well the phone works. iPhone simply works better. Since software and hardware are more married, the experience is more consistent. In my own experience, apps crash less on iPhone. The phone lags less. Scrolling and browsing is more graceful. My wife is still on her original Droid for another month, and every day I see her struggle with the phone.

iPhone has better customer satisfaction ratings than Android. While the reasons aren't mentioned, I wouldn't be surprised that part of it stems from "it just works".

Apple has a strong emphasis on usability. Google and the OEMs aren't as much so. This particularly stuck me with a post I read about a guy talking to a girl who had an Android phone and an iPod Touch. In particuar, "nothing happened when I plugged it into my computer." To the average user, the simple integration is important. They don't know why the phone shows up as a "Mass Storage Device" when they plug it in.

Future

The future for Android will probable improve. The OS is maturing and hardware getting better, however the ecosystem of manufacturers and carriers will likely stay the same. The reality is that Apple is doing what Apple has always done and they've gotten really good at it.

Cooking with Chef slides

Mon Jul 25 00:00:00 -0700 2011

A little late, but I have posted my slides from my talk at the East Bay Ruby Meetup in June.

Cooking with Chef

Check them out and feel free to ping me if there are any questions. At the meetup, was also talking about doing a bit of a blog series on getting started with Chef and posting some of the scripts and baseline setup I have used before. Hope to start forming some simple getting started resources.

New Opportunities

Tue Jul 19 00:00:00 -0700 2011

Finally posting about it a little bit late, but last week was my first week in a new position with a new company. I truly enjoyed my stay at Involver, but saw an opportunity pop up and decided to take it.

This new opportunity is to be one of the first in-house Ruby developers at Demandbase. Demandbase specializes in a B2B analytics API for reporting and logging information about visitors hitting your business's website.

That all sounds fancy, but the gist is I'll be helping to grow and form the team, scale up and expand the platform, and able to leverage both skillsets as a developer and in operations. They've been working with the awesome guys from Highgroove, utilizing Chef in a continuous deployment setup, doing geographic load balancing, and have a pretty agile and responsive process setup.

I had an amazing time at Involver and grew so much personally and professionally. I was a part of some major projects and instrumented a lot of change. Our Operations team was top notch and put out of a lot of stuff considering it was only 3 people.

Going forward, have some goodies I'll hopefully be working on and open sourcing. There are a lot of interesting things in the works and hope to post about them over time.

And by the way, Demandbase is hiring. Hit me up @krobertson or shoot me an email.

Announcing Gemstache

Thu Jun 30 00:00:00 -0700 2011

Pleased to announce Gemstache, a new service to enable teams and companies to build and distribute their own private gems with security and ease of use in mind.

Gems are an incredibly simple way to package up code and distribute it, making it available to multiple code bases or just a great way to encapsulate functionality. However, the way gems are traditionally distributed doesn't lend itself well to securing the gems and controlling access. The default "gem install" command you run doesn't provide any support for authentication against the gem source, so most sources are open to the public.

Gemstache works by giving you your own private gem source you can use. In order to download any gems from it, you must have the accompanied "gemstache" gem installed locally or on your servers. When talking to your private gem source, it interlaces to add authentication to the request as well as ensuring it is happening over HTTPS. With this, your gems are only downloaded over SSL and any interaction requires authentication.

To make managing your gems easier, the gem adds the ability to easily upload a gem to your private gem source and make it available. From your command line, just run "gem stache [your gem file]" and it is uploaded and ready for use.

Within the service, you can give users varying roles to define whether they can just download gems, are able to publish new gems, and are able to yank/delete gems.

Gemstache was born out of a need we had been facing at Involver, where some of our own internal code is enclosed in gems and used between a few of our codebases, as well as we have some gems we have forked to either add customizations or our own fixes. Traditional methods left them exposed and also more painstaking to release.

We're just about ready to launch a public beta. Visit gemstache.com and sign up to be notified.

Beware of error handling with fibers in an evented context

Thu May 26 00:00:00 -0700 2011

One of the more interesting developments in Ruby lately has been the interesting things being done with Ruby 1.9.2, Fibers, and EventMachine. Whether you look at it being applied to existing libraries with rack/fiber_pool, or Goliath, it packs a punch and proving interesting for high performance evented applications.

Recently when reworking one of our backend applications to use rack/fiber_pool with EventMachine, I ran into a gotcha when it came to exception handling. Asynchronous applications makes heavy use of callbacks, where the context of when an operation is completed is not inline with when it began. Fibers work to make it easier to write asynchronous code look like synchronous, where you make a call, and it returns a result.

The problem is that this can play a trick on your eye. You think you are executing something in line, but the context it is operating within jumps without you being necessarily aware of it. Take the following example:

require 'eventmachine'
require 'em-synchrony'
require 'fiber'

def doit
  f = Fiber.current
  EventMachine.add_timer(1) { raise 'uhh ohh' }
  Fiber.yield
end

EM.synchrony do
  begin
    puts "beginning"
    doit
    puts "end"
  rescue Exception => e
    puts "ERROR: #{e.message}"
  ensure
    puts "Have a nice day"
  end
  EventMachine.stop
end

From looking at the main block, you have the call to doit wrapped in a begin/rescue, so it seems logic that if an error was raised within it, the rescue block would capture it and handle the error. Unfortunately, the error it being raised outside the context of the fiber, so it isn't caught. You are outside the context of the fiber in the time between when Fiber.yield is called and f.resume is called.

Why is this context important? Because when we built applications, we want them to be resilient. For instance, if a web request encounters an error, we want it caught, reported either to a log or a service, a response returned to the user, and then it be ready to handle the next request. However outside the context of the fiber, the error handling we've built in won't get called and the application can exit. Even if the application is monitored with god, existing connections will get closed out and connections won't be served while it is restarting, and the valuable information within the error is lost.

Once the fiber is resumed, the error handling functions as expected.

require 'eventmachine'
require 'em-synchrony'
require 'fiber'

def doit
  f = Fiber.current
  EventMachine.add_timer(1) { f.resume }
  Fiber.yield
end

EM.synchrony do
  begin
    puts "beginning"
    doit
    raise 'uhh ohh'
  rescue Exception => e
    puts "ERROR: #{e.message}"
  ensure
    puts "Have a nice day"
  end
  EventMachine.stop
end

In this case, the exception will be caught and handled, and you will make it to the "Have a nice day" message.

In order to handle the error, you can add some exception handling within the main context, or the root fiber.

begin
  EM.synchrony do
    begin
      puts "beginning"
      doit
      puts "end"
    rescue Exception => e
      puts "ERROR: #{e.message}"
    ensure
      puts "Have a nice day"
    end
    EventMachine.stop
  end
rescue Exception
  puts "We caught something outside a fiber!"
end

However, I haven't yet figured out how to have the same thing done in the context of a web app. So far, I've just ensured I have proper error handling in the callbacks before the fiber is resumed. Trying to put a begin/rescue block in the config.ru doesn't seem to have any effect on rescuing an error.

The moral of the story is when using fibers, be conscious of error handling outside the context of the fiber. Unhandled exceptions there can cause the entire application to exit. Be aware you have proper error handling when doing something that yields a fiber, and whether they fiber-enabled libraries you are using have error handling outside the context as well.

MySQL Read/Write Splitting in JRuby

Mon May 16 00:00:00 -0700 2011

I was rather surprised about the lack of information out there about how to do read/write splitting with MySQL and Rails on JRuby. This is a pretty key part of our infrastructure, and has been a major point of our attention when performing large platform upgrades.

Having recently gone through one such upgrade and subsequent breaking of our read/write splitting, I had to dive into the code that was doing it more and get a better understanding of how it works.

With JRuby, there are two things to be aware of when trying to enable read/write splitting:

Does it route the queries to the right box?
Does it do it without sucking all available connections up on MySQL?

The second one sounds weird, however we have actually seen it and had it cause major issues. I am not totally sure what to attribute it to, but somewhere between ActiveRecord, ActiveRecord-JDBC, and the MySQL JDBC driver, there is some weird connection handling going on where a connection will still be established, however viewed as dead and a new connection will be opened up. Eventually, it just sucks up all available connections to MySQL.

This is a pretty big problem, but can work around it. Most of the existing information out there about how to do read/write splitting with ActiveRecord doesn't always apply cleanly to JRuby. In particular with one, we ran into the connection-sucking issue while in the benchmarking process.

Fortunately, there isn't a whole lot that needs to be done to enable splitting. The MySQL JDBC driver already has support for it with its ReplicationDriver, you just need to enable it when using Rails. Then you can leverage a plugin which sets a "read_only" property on the raw connection when it recognizes a 'SELECT' query. When read_only is flagged, the driver knows it can send the query to servers other than the master.

Steps to configure read/write splitting:

1) Get activerecord-jdbcmysql-adapter

If you're going to use MySQL on JRuby with ActiveRecord, this is rather a given. We're currently using v1.1.1 and happy. All the JDBC adapters are basically rolled up in one repo which produces multiple gems.

2) Get my version of the active-record-jdbc-mysql-master-slave plugin

There is a plugin that takes care of setting the read_only property on the connection for 'select' queries. The original version of the plugin was originally written for Rails 2.2 and hasn't been updated in 2 years. We originally used it on Rails 2.2.2 and had a lot of success with it.

It doesn't work on Rails 2.3 through, primarily because of the load order changing. It previously worked by alias_methoding the initializer on JdbcAdapter, and in its initialize, would alias_method the _execute method if using mysql. However, load order changed in Rails 2.3 and the database connection is initialized before the plugin is loaded, causing its own version of initialize to not run. I've updated it to use a slightly different method to ensure it happens regardless of load order.

To install the plugin, can just use ./script/plugin:

$ ./script/plugin install git://github.com/krobertson/active-record-jdbc-mysql-master-slave.git

3) Ensure active connections are reset in ActiveRecord

You can still experience some issues with stray connections. The best way we have found to combat this is through an after filter on controllers that does some cleanup. Add this portion to your ApplicationController:

after_filter  :clear_database_connections

def clear_database_connections
  ActiveRecord::Base.clear_active_connections!
end

By calling ActiveRecord::Base.clear_active_connections!, it ensures connections are reset. It is best to try and ensure it is the last after_filter defined.

4) Configure your database.yml

The key in the database.yml is to set up activerecord-jdbc to use MySQL's replication driver and to configure the "url" for it.

production:
  adapter:  jdbcmysql
  username: myuser
  password: secret
<% if $0 =~ /(rake|irb)/ -%>
  host:     masterdb
  port:     3306
  database: myapp
<% else -%>
  driver:   com.mysql.jdbc.ReplicationDriver
  url:      jdbc:mysql://masterdb:3306,slavedb:3306/myapp?roundRobinLoadBalance=false&autoReconnect=true&failOverReadOnly=true
<% end -%>

You may be looking at it and thinking I've got erb in my yaml... and yes. Rails lets you mix the two, and the reason in this case is because we opt to disable the read/write splitting in the Rails console (irb) and when running rake. Two very important reasons to. First, in console, we've had it suck up all connections just when doing simple debugging. Not fun. The normal adapter doesn't do it. And second, we want it off in rake for when we deploy and it does db:migrate. We don't want to allow replication delay in there, such as when querying the schema_migrations table.

For the read/write splitting part, the bulk of the settings are in the "url". The first host specified is the master, and any others are slaves. It is supposed to support multiple slaves and have the option to load balance between them, though we found in practice it never really worked. Instead we direct our web app and background processing to separate slaves. After the hosts comes the database name, and the query string parameters are a number of the driver options.

How the stuff works

At first, looking at the code for the plugin seemed rather confusing. Less than 100 lines of injecting into ActiveRecord did our read/write splitting? But the truth is it is fairly simple, as most of the work is handled by the underlying driver (MySQL's JDBC driver). All we really need to do is populate that read_property on the driver's connection. Shortened up, it does this:

alias_method :_execute_without_master_slave, :_execute
alias_method :_execute, :_execute_with_master_slave

# if we're in auto-commit mode and about to execute a select statement, 
# then set the connection in read-only mode for the duration of
# the query... which will permit the query to be load-balanced
# amongst the slaves by the mysql connector/j ReplicationDriver
def _execute_with_master_slave(sql, name=nil)
  # Need to set the read_only option on the raw connection
  # to tell the underlying driver whether the request can
  # go to slaves.
  cro = raw_connection.connection.read_only
  begin
    raw_connection.connection.read_only = 
      raw_connection.connection.auto_commit &&
      JdbcConnection::select?(sql)

    self._execute_without_master_slave(sql, name)
     
  ensure
    raw_connection.connection.read_only = cro
  end
end

To sum it up, the adapter has an _execute method which does the real execution and is passed in the raw SQL. The plugin injects itself in the middle. First, it captures the current read_only value, because after it executes the given query, it wants to ensure it is set back to that. Next, it sets the read_only property based on whether MySQL itself has auto_commit on and if it is a 'select' query. It then calls the original _execute method, and after it runs, sets read_only back.

The actual plugin has several additional things, like allowing you to run some statements within a block to ensure it goes to the master, and to only work with the MySQL adapter.