In Valid Logic

Deploying Ubuntu 12.04 on XenServer Made Easy

2012-05-01T00:00:00-07:00

To follow up on my previous post about disk errors with Ubuntu 12.04 on XenServer, I wanted to cover the process I've put in place for provisioning Ubuntu VMs.

With PaaS.io, I have a mixture between systems deployed on bare metal and virtualized. With the virtualized systems, I set out to make the provisioning as easy as if I was using an IaaS provider, while still giving me fine control over sizing and placement. Some of the decisions there are enough for another post.

Ubuntu 12.04 is the new hotness and I'd been anxiously awaiting it, with plans to progressively roll it out throughout PaaS.io.

With XenServer, it provides an easy template for provisioning Ubuntu Lucid 10.04 VMs and even for some newer releases. However, the way XenSever provisions Ubuntu is to essentially netboot it and install it from a remote source. Because of that, we can't simply use one of those templates buts install a newer release.

Fortunately, the template itself is very simple, and which release it installs is just a configuration parameter.

To create a new Ubuntu 12.04 template, simply log into the XenServer console and runt he following commands. It will clone the Lucid template and then change the parameter for the release to install from lucid to precise.

$ TEMPLATE_UUID=`xe template-list \
        name-label="Ubuntu Lucid Lynx 10.04 (64-bit)" params=uuid --minimal`
$ NEW_TEMPLATE_UUID=`xe vm-clone uuid=$TEMPLATE_UUID \
        new-name-label="Ubuntu Precise (64-bit)"`
$ xe template-param-set other-config:default_template=true \
        other-config:debian-release=precise uuid=$NEW_TEMPLATE_UUID

Now you will find a "Ubuntu Precise (64-bit)" option in the template list of XenCenter.

Now, we can actually provision a new box. Next, one interesting discovery was that you can pass in a kickstart script in the boot parameters options for the new VM.

Kickstart allows you to do a scripted installation, automated pretty much every aspect of the system. Instead of picking a bunch of options, it allows for an easy, repeatable process that basically leaves the machine completely ready to go.

To use a kickstart script, make a script available over HTTP somewhere on your LAN or on the public internet. By the time it is grabbed, the machine will have an IP, DNS, and all. Then, simply add ks=http://url/to/your/kickstart to the beginning part of "advanced OS boot parameters" option when selecting the installation media.

Below is a cleaned version of the kickstart script I used on my Ubuntu VMs. The main things of note:

Sets up my partition table
Configures base system with ubuntu-minimal as well as some common packages like openssh-server, curl, wget, and screen.
Disables the creation of the ubuntu user (I create a normal every day user through chef)
Configures the fstab with barrier=0 as mentioned before
Disables /bin/sh from pointing to /bin/dash (personal preference)
Updates apt sources
Installs xenstore-utils
Downloads some auto-configuration scripts
Installs XenTools
Installs the Ubuntu virtual kernel and removes the generic one
Cleans up apt caches
Shuts down the VM.

lang en_US
langsupport en_US
keyboard us
timezone America/Los_Angeles
text
install
skipx
halt
url --url http://us.archive.ubuntu.com/ubuntu

rootpw pa$$word   # you should replace, and use --iscrypted
auth --useshadow --enablemd5
user --disabled

bootloader --location=mbr
zerombr yes
clearpart --all --initlabel
part /boot --fstype=ext2 --size=64
part swap --size=1024
part / --fstype=ext4 --size=1 --grow

network --device=eth0 --bootproto=static --ip=10.0.0.50 --netmask=255.255.255.0 \
        --nameserver=10.0.0.1 --gateway=10.0.0.1
firewall --disabled

%packages
ubuntu-minimal
openssh-server
screen
curl
wget

%post

# update fstab for the root partition
perl -pi -e 's/(errors=remount-ro)/noatime,nodiratime,$1,barrier=0/' /etc/fstab

# point sh to bash instead of dash
rm /bin/sh
ln -s /bin/bash /bin/sh

# add normal apt source list
(
cat <<'EOP'
deb http://us.archive.ubuntu.com/ubuntu/ precise main restricted universe
deb http://us.archive.ubuntu.com/ubuntu/ precise-security main restricted universe
deb http://us.archive.ubuntu.com/ubuntu/ precise-updates main restricted universe
EOP
) > /etc/apt/sources.list
apt-get update
apt-get upgrade -y

# install some additional packages
apt-get install -y xenstore-utils

# set up xenserver automation scripts
AUTOMATER_REPO=https://raw.github.com/krobertson/xenserver-automater
curl $AUTOMATER_REPO/master/usr/sbin/xe-set-hostname > /usr/sbin/xe-set-hostname
curl $AUTOMATER_REPO/master/usr/sbin/xe-set-network > /usr/sbin/xe-set-network
curl $AUTOMATER_REPO/master/usr/sbin/generate-sshd-keys > /usr/sbin/generate-sshd-keys
curl $AUTOMATER_REPO/master/etc/init/xe-automator.conf > /etc/init/xe-automator.conf
chmod a+x /usr/sbin/xe-set-hostname
chmod a+x /usr/sbin/xe-set-network
chmod a+x /usr/sbin/generate-sshd-keys

# setup locales
locale-gen en_US.UTF-8
update-locale LANG="en_US.UTF-8"
echo 'LANG=en_US.UTF-8' >> /etc/environment
echo 'LC_ALL=en_US.UTF-8' >> /etc/environment

# install xe tools
cd /tmp
wget http://some/url/to/xe-guest-utilities_6.0.0-743_amd64.deb
dpkg -i xe-guest-utilities_6.0.0-743_amd64.deb

# install paravirt kernel image
apt-get install -f -y linux-virtual
dpkg -l | grep generic | grep linux | awk '{print $2}' | xargs apt-get remove -y

# clean up some stuff
rm -f /etc/ssh/ssh_host_*
rm -f /var/cache/apt/archives/*.deb
rm -f /var/cache/apt/*cache.bin
rm -f /var/lib/apt/lists/*_Packages

It is important that it shuts down at the end. My goal was to have it be all inclusive, and that means setting up the virtual kernel. However, it can't successfully reboot, because we need to update some PV boot options before it is ready to go.

Also, some packages must be installed in the post steps. The default install sources for Ubuntu don't always have all packages available, and I found it best to do the kernel last.

When you first bring up the new VM in XenServer, you may need to enter in a few details if you don't have DHCP running. It will self configure by default, but I normally opt to manually configure it to ensure the template gets a certain IP just to avoid any future possible collision.

After the bootstrapping is done and the VM is then off, log into console on the XenServer host itself and run the following snippet:

$ VMNAME=precise-20120501
$ UUID=`xe vm-list name-label="$VMNAME" params=uuid --minimal`
$ EDITOR=cat xe-edit-bootloader -n "$VMNAME" -p 1 \
        -f /grub/grub.cfg > /tmp/$VMNAME-grub
$ KERNEL=`grep vmlinuz /tmp/$VMNAME-grub | grep virtual |
        grep -v recovery | awk '{print $2}'`
$ ROOT=`grep vmlinuz /tmp/$VMNAME-grub | grep virtual |
        grep -v recovery | awk '{print $3}'`
$ RAMDISK=`grep initrd /tmp/$VMNAME-grub | head -1 | awk '{print $2}'`
$ xe vm-param-set uuid=$UUID PV-bootloader-args="--kernel=$KERNEL --ramdisk=$RAMDISK"
$ xe vm-param-set uuid=$UUID PV-args="$ROOT ro quiet console=hvc0"

Before you run it, of course set the VMNAME to the name of your VM. You might also want to check some of the values as you go (echo $UUID). You can use the xe-edit-bootloader command to view the grub configuration within the VM, but need to set the PV settings outside the guest. By faking the EDITOR to cat, you can export the grub file to a local file, then use some grep+awk to get the necessary pieces and finally set the PV settings correctly.

At this point, the machine is ready to be booted, converted into a template, or simply cloned. I personally like to leave my templates as never-been-used.

One of the main benefits of an IaaS setup like OpenStack or CloudStack is the self orientating of the VMs. Normally with a template, it keeps the template's setting for its hostname and network configuration. However, I found some example scripts on Github for passing network information into a VM through the VM's "xenstore" options. That way on boot, the VM can read in the settings it needs and update itself. I did some work to update the scripts to work with Precise, to more thoroughly update the hostname and DNS, as well as to regenerate the ssh host keys (since the kickstart did delete them).

As an example, this would be how to clone the template to a VM, set the params, and boot it:

$ UUID=`xe vm-install template=precise-20120501 new-name-label=builder`
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/ip=10.0.0.11
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/gw=10.0.0.1
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/nm=255.255.255.0
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/ns='10.0.0.1 10.0.0.2'
$ xe vm-start uuid=$UUID

That is cool and all, but I don't want to log into the XenServer console each time I want to setup a VM. Luckily, found a plugin for Chef's knife utility that adds XenServer provisioning. I forked it and added setting xenstore network parameters to it.

Now, provisioning a new VM, configuring it, and bootstrapping chef on it is just a matter of one call:

$ knife xenserver vm create --vm-template precise-20120501 \
                            -x root --keep-template-networks \
                            -r "role[foo]" -E production \
                            --vm-ip 10.0.0.12 --vm-name foobar

The command may be a little long, but it is all encapsulated in a single command. I simply have an internal README of sorts with the command pre-prepared for various roles.

If you have any questions, please leave a comment and ask. I'd be glad to help and always looking to improve my process as well.

Fixing random disk errors with Ubuntu 12.04 Precise on XenServer

2012-04-28T00:00:00-07:00

Ubuntu 12.04 LTS Precise is hot off the press right now. Over the past few days, have been working on building new base images for PaaS.io, but was running into random issues where the root parition would encounter an error and freeze up. It normally happened just after it would finish booting and about 50% of the time.

A few times it happened after I already logged in, so was able to do basic read only operations. In dmesg, was able to see the following:

[    6.748868] blkfront: barrier: empty write xvda op failed
[    6.748876] blkfront: xvda: barrier or flush: disabled
[    6.748890] end_request: I/O error, dev xvda, sector 6584768
[    6.748908] end_request: I/O error, dev xvda, sector 6584768
[    6.748943] Aborting journal on device xvda6-8.
[    6.767022] EXT4-fs error (device xvda6): ext4_journal_start_sb:327: Detected aborted journal
[    6.767046] EXT4-fs (xvda6): Remounting filesystem read-only

Or if you weren't able to log in first, you might see this in the XenCenter console:

After some Googling, was able to track a similar error down by someone else. The fix was to update the mount options for the root partition. Mine are now:

noatime,nodiratime,errors=remount-ro,barrier=0

The key is the barrier=0. From some documentation, it is an option to help increase the integrity of writes by ensuring everything is flushed to disk be committing to the journal. However sometimes in a virtualized environment that is difficult to guarantee. In my case, have disk->RAID->dom0->LVM->domU.

Figure many other people will be diving into Precise this weekend, potentially running into this issue like me.

Soon, will post some additional details about how to easily get a nice Precise template in XenServer 6. I've been getting my setup nicely tuned using a kickstart script for the base system and leveraging xenstore data to dynamically setup the IP and hostname on boot.

Configuring MongoDB Replica Sets With Keepalived

2012-02-23T00:00:00-08:00

While working on developing PaaS.io, one of my primary objectives is to ensure that everything is configured to be truly Highly Available (HA). This means everything has a secondary, replicated, and has automated failover. This way it is nicely resilent and fault tolerant. Core infrastructure changes can be made without any affect to running applications, servers go down without issues. Sweet, huh?

When it came to setting up MongoDB, the obvious option was to go to using Replica Sets. They're like master-slave replication in RDBMSes but the master node is essentially floating. Client applications connect to the whole cluster, identify the master, and then start to speak to it.

By default, Cloud Foundry provisions a dedicated MongoDB instance for you and provides you the IP, port, and other credentials. This doesn't include replication, and focuses on single IP connections rather than replica sets.

However when using Replica Sets, the connection pattern in your code is slightly different. You use a ReplSetConnection instead of a normal Connection, in ruby driver speak. So you need to know you're connecting to a Replica Set if the host you're connecting to isn't the master. If you only connect to the master though, you can use a traditional connection.

I was interested in maintainin full compatibility with Cloud Foundry's out-of-the-box experience, so first looked at using mongos in front of replicas. It is normally used to front a sharded setup, but using it without sharding is still a feature request.

So I set out looking at how to use a virtual IP have it track which node is the master. In this case, if one switches over, it will pick it up and move the virtual IP to follow. This would be mostly transparent to the application. The only handling in the app is the next query will fail, but quickly reconnect and its fine.

I'd already been using keepalived for HA within HAProxy, and one of the nice things it provides is the ability to define a script to run as a part of its local health checks. Using this, can have a script that just asks the local system "you the master?" and returns appropriately.

Below is the script:

#!/usr/bin/env ruby

require 'optparse'

options = { :hostname => '127.0.0.1',
            :port     => 27017,
            :username => nil,
            :password => nil }

optparser = OptionParser.new do |opt|
  opt.banner = "Usage: #{$0} [options]"
  opt.on('-H', '--hostname HOST', 'Hostname') { |o| options[:hostname] = o if o }
  opt.on('-P', '--port PORT', 'Hostname') { |o| options[:port] = o.to_i if o }
  opt.on('-u', '--username USER', 'Username') { |o| options[:username] = o if o }
  opt.on('-p', '--password PASSWORD', 'Password') {|o| options[:password] = o if o }
end

begin
  optparser.parse!
rescue => e
  $stderr.print e
  $stderr.print optparser
  exit 1
end

require 'rubygems'
require 'mongo'

conn = Mongo::Connection.new(options[:hostname], options[:port])
conn.db('admin').authenticate(options[:username], options[:password])
master_status = conn.db('admin').command({'isMaster'=>1})

if master_status['ismaster']
  exit 0
else
  exit 1
end

This won't return any output, but will exit with 0 if it is the master or 1 if it isn't.

Its requirements are simple... Ruby, RubyGems, mongo gem, and recommend the bson_ext gem too.

Then within our keepalived.conf file, it looks like this:

global_defs {
   notification_email {
     me@example.com
   }
   notification_email_from system@example.com
   smtp_server 192.168.1.1
   smtp_connect_timeout 30
}

# Define the script used to check if mongod is running
vrrp_script chk_mongod {
    script "killall -0 mongod"
    interval 2 # every two seconds
    weight 2
}

# Define the script to see if the local node is the primary
vrrp_script chk_mongo_primary {
    script "/usr/local/mongodb/bin/mongo_check_primary -u admin -p password"
    interval 2 # every two seconds
    weight 2
}

# Configuation for the virtual interface
vrrp_instance VI_1 {
    interface bond0
    state node MASTER        # SLAVE on the other nodes
    priority 101             # 100 on other nodes
    virtual_router_id 55

    authentication {
        auth_type AH
        auth_pass secret     # Set this to some secret phrase
    }

    # The virtual ip address shared between the two nodes
    virtual_ipaddress {
        192.168.1.222
    }

    # Use the script above to check if we should fail over
    track_script {
        chk_mongod
        chk_mongo_primary
    }
}

With this in place, if a server goes down, it'll switch. If mongod dies, it will switch over (since Mongo itself will recognize that), and if the current primary is switched, the mongo_check_primary will return an exit code of 1, causing it to switch over.

Another reason this might be useful in some cases is because some clients still don't support replica sets. With this method, they don't need to.

In addition to these steps, you will need to perform the normal steps to setup keepalived. These would include:

Ensure mongod is bound to 0.0.0.0 so it'll accept any incoming connection.
Set net.ipv4.ip_nonlocal_bind=1 in sysctl so you can use virtual IPs.

Try this out and let me know your experiences. If you're interested in this kind of seamless infrastructure automation, check out PaaS.io or drop me a line.

Jekyll on PaaS.io with Cloud Foundry

2012-01-06T00:00:00-08:00

Recently I moved my blog over to the service I am currently working on building, PaaS.io.

Running Jekyll on PaaS.io isn't all that different from running it on other services, though I had a few other goals in mind. The way I wanted it set up was:

Stop using rack-jekyll. Its a nice gem, however it is locked to an older version of Jekyll. And the current gem is on an even more outdated one. Currently, Cloud Foundry doesn't support pulling bundler sources from git too.
Have a public folder for static content like CSS and images.
Have the _site folder for generated content
Don't have it copy the Gemfile and the config.ru into the _site folder (annoys me)
Redirect www.invalidlogic.com to invalidlogic.com
Low foot print

First, the Gemfile:

source :rubygems

gem 'rack-contrib', :require => 'rack/contrib/try_static'
gem 'rack-redirect'
gem 'thin'

group :development do
  gem 'jekyll'
  gem 'RedCloth'
  gem 'rdiscount'
end

The main gems being used are thin, rack-contrib (for TryStatic, note on that later), and rack-redirect (for www redirection). I also include some of the gems I use for Jekyll in the development group. That way they are available locally but not loaded when I deploy (lower footprint... and yes, it is minor).

Now, the config.ru:

require 'rubygems'
require 'bundler'
Bundler.require

use Rack::EY::Solo::DomainRedirect

use Rack::TryStatic,
    :root => "_site",
    :urls => %w[/],
    :try => ['.html', 'index.html', '/index.html']

use Rack::Static,
    :root => "public",
    :urls => %w[/]

run lambda { [404, {'Content-Type' => 'text/html'}, ['Not Found']]}

There are 4 rack components here. First, Rack::EY::Solo::DomainRedirect is the rack-redirect gem and handles the www redirection. Next, is Rack::TryStatic. It is used to access files from the _site generated content directory. It gives a couple different :try values for different ways to find the intended file. Then is the Rack::Static which gets static content from the public directory. No need to try different combinations. And last is a generic lambda that will return 404.

Next, want to avoid duplication. With things as they are, when I run jekyll it will copy the public and other items into the _site folder duplicating it. To resolve that, in our _config.yml, can add an exclude line:

exclude: [ 'public', 'Gemfile', 'Gemfile.lock', 'config.ru' ]

And with that, we are set! All of our goals are met. Commit and push to deploy! Currently I have Cloud Foundry set up with a Rack framework defined (will be sending a pull request with it soon) and also have my blog set to use Ruby 1.9.3 as well.

Soon I'll be providing some more details on PaaS.io, so stay tuned and click over to it and sign up to get access to the beta.

iPhone vs Android is the new Mac vs PC

2011-10-18T00:00:00-07:00

After spending 18 months with Android, I am now back to an iPhone and likely here to stay. While Android isn't necessarily bad, it more boils down to iPhone simply being better. After a while, I started looking at iPhone vs Android as largely a repeat of the Mac vs PC comparisons.

Mac vs PC

Just look at it. Apple is doing what they've always done. They control the hardware and the software. They have a solid, unified experience. You can pick up any iPhone, old or new, and still be at home.

Android is repeating the history of PCs. Google makes the OS, as Microsoft did with Windows. Then OEMs offer a wide variety of different hardware, with their own shitty customizations layered on top of the OS. Even worse, you have the carriers layering on their customizations and restrictions.

You can go from one Android phone to another and have it be completely foreign. Even baseline apps can have different names and a completely different look and feel, such between the "Email" vs "Mail" of vanilla Android and HTC Android.

Shelf Life

With Android, the phones have a very short shelf life. I bought a Thunderbolt from Verizon back in April, just 6 months ago. About a month after I bought it, it was no longer the hot model they were pimping. In fact, there have been 2-3 phones to come out since then that became "king of the mountain."

With iPhone, they release a new phone about once a year, and that one stays the current phone. Older phones are still pretty well supported. The iPhone 3GS is over 2 years old, still got updated to iOS 5, and likely will until iOS 6. My original Motorola Droid that is nearing 2 years old is pretty much forgotten already.

Updates with iPhone? Available right away to everyone. AT&T or Verizon, you get the update when Apple makes it generally available.

Updates with Android? Have fun. Google has to release it, your OEM has to customize it, then the carrier gets to tweak it, and decide when to finally roll it out. Google released Gingerbread back in December 2010, nearly 10 months ago. Verizon just started rolling out the update last month, then pulled it. Even then, they decide when you can upgrade with a phased roll out. And since it was pulled, looks like they seem to skip over adequate testing.

Most Android users who want recent releases end up rooting their phones and use unofficial ROMs put together by an informal group of people. Have an issue with your phone? Limited options.

Marketing

I definitely agreed with my friend Joe in his post about the Droid Bionic. Android phones are being pitched like PCs. They give a bunch of technical specs that are meaningless to consumers. It echos the "Golden Circle" idea by Simon Sinek. Apple is still doing what they do best. Verizon sells like PC manufacturers. Lacking inspiration.

False Sense of Market Share

You've probably seen the headlines of "Android sales outpacing iPhone" or "Android market share to surpass iPhone".

Overall the numbers are comparing apples to oranges. They are comparing the sales figure of a single phone to a whole group of phones. There are many different Android phones, and individually, iPhone is spanking them in sales figures. No single Android phone has a chance of elipsing the iPhone or gaining any meaningful market share, especially with their overall short shelf life. I'd be very interested to see sales figured of individual phones and see how long they really last. All the manufacturers are still struggling to get a sliver of what the iPhone is capable of.

The numbers may hold some weight for developers because it represents the size of your audience. But at the same time, Android is many phones vs iPhones entire existence is now just 5 phones. Less "your app doesn't work on my Verizon Droid Mumbojumbo" and you don't have a Droid Mumbojumbo to test on.

Falseness of Open

Many people tout Android as being open, however the actions of Google with Honeycomb's source code seem to be heading in the direction of more closed.

While Google said part of their goal was to try and unify the platform more, as Honeycomb was intended for tablets and not handsets, the ability to control a platform is difficult while also keeping it "open." I think Google is right in closing it, since in order to further the platform, they will need to have some control in order to maintain a consistent direction.

However, the "openness" usually just comes from developers. What do most of then define the "openness" as? Being able to write apps and put them on their phone without paying $99. They tout the source being open, but the truth is that isn't what they really care about. Very few Android developers likely dive into the OS code, they just want to install their own apps for free.

Personally, if I prefer one platform over the other, a $99 fee to build apps isn't going to a stopper for me. But I also don't mind paying for the tools I use in my craft if they are worth it.

Working

Most average people care more about how well the phone works. iPhone simply works better. Since software and hardware are more married, the experience is more consistent. In my own experience, apps crash less on iPhone. The phone lags less. Scrolling and browsing is more graceful. My wife is still on her original Droid for another month, and every day I see her struggle with the phone.

iPhone has better customer satisfaction ratings than Android. While the reasons aren't mentioned, I wouldn't be surprised that part of it stems from "it just works".

Apple has a strong emphasis on usability. Google and the OEMs aren't as much so. This particularly stuck me with a post I read about a guy talking to a girl who had an Android phone and an iPod Touch. In particuar, "nothing happened when I plugged it into my computer." To the average user, the simple integration is important. They don't know why the phone shows up as a "Mass Storage Device" when they plug it in.

Future

The future for Android will probable improve. The OS is maturing and hardware getting better, however the ecosystem of manufacturers and carriers will likely stay the same. The reality is that Apple is doing what Apple has always done and they've gotten really good at it.

Cooking with Chef slides

2011-07-25T00:00:00-07:00

A little late, but I have posted my slides from my talk at the East Bay Ruby Meetup in June.

Cooking with Chef

Check them out and feel free to ping me if there are any questions. At the meetup, was also talking about doing a bit of a blog series on getting started with Chef and posting some of the scripts and baseline setup I have used before. Hope to start forming some simple getting started resources.

New Opportunities

2011-07-19T00:00:00-07:00

Finally posting about it a little bit late, but last week was my first week in a new position with a new company. I truly enjoyed my stay at Involver, but saw an opportunity pop up and decided to take it.

This new opportunity is to be one of the first in-house Ruby developers at Demandbase. Demandbase specializes in a B2B analytics API for reporting and logging information about visitors hitting your business's website.

That all sounds fancy, but the gist is I'll be helping to grow and form the team, scale up and expand the platform, and able to leverage both skillsets as a developer and in operations. They've been working with the awesome guys from Highgroove, utilizing Chef in a continuous deployment setup, doing geographic load balancing, and have a pretty agile and responsive process setup.

I had an amazing time at Involver and grew so much personally and professionally. I was a part of some major projects and instrumented a lot of change. Our Operations team was top notch and put out of a lot of stuff considering it was only 3 people.

Going forward, have some goodies I'll hopefully be working on and open sourcing. There are a lot of interesting things in the works and hope to post about them over time.

And by the way, Demandbase is hiring. Hit me up @krobertson or shoot me an email.

Announcing Gemstache

2011-06-30T00:00:00-07:00

Pleased to announce Gemstache, a new service to enable teams and companies to build and distribute their own private gems with security and ease of use in mind.

Gems are an incredibly simple way to package up code and distribute it, making it available to multiple code bases or just a great way to encapsulate functionality. However, the way gems are traditionally distributed doesn't lend itself well to securing the gems and controlling access. The default "gem install" command you run doesn't provide any support for authentication against the gem source, so most sources are open to the public.

Gemstache works by giving you your own private gem source you can use. In order to download any gems from it, you must have the accompanied "gemstache" gem installed locally or on your servers. When talking to your private gem source, it interlaces to add authentication to the request as well as ensuring it is happening over HTTPS. With this, your gems are only downloaded over SSL and any interaction requires authentication.

To make managing your gems easier, the gem adds the ability to easily upload a gem to your private gem source and make it available. From your command line, just run "gem stache [your gem file]" and it is uploaded and ready for use.

Within the service, you can give users varying roles to define whether they can just download gems, are able to publish new gems, and are able to yank/delete gems.

Gemstache was born out of a need we had been facing at Involver, where some of our own internal code is enclosed in gems and used between a few of our codebases, as well as we have some gems we have forked to either add customizations or our own fixes. Traditional methods left them exposed and also more painstaking to release.

We're just about ready to launch a public beta. Visit gemstache.com and sign up to be notified.

Beware of error handling with fibers in an evented context

2011-05-26T00:00:00-07:00

One of the more interesting developments in Ruby lately has been the interesting things being done with Ruby 1.9.2, Fibers, and EventMachine. Whether you look at it being applied to existing libraries with rack/fiber_pool, or Goliath, it packs a punch and proving interesting for high performance evented applications.

Recently when reworking one of our backend applications to use rack/fiber_pool with EventMachine, I ran into a gotcha when it came to exception handling. Asynchronous applications makes heavy use of callbacks, where the context of when an operation is completed is not inline with when it began. Fibers work to make it easier to write asynchronous code look like synchronous, where you make a call, and it returns a result.

The problem is that this can play a trick on your eye. You think you are executing something in line, but the context it is operating within jumps without you being necessarily aware of it. Take the following example:

require 'eventmachine'
require 'em-synchrony'
require 'fiber'

def doit
  f = Fiber.current
  EventMachine.add_timer(1) { raise 'uhh ohh' }
  Fiber.yield
end

EM.synchrony do
  begin
    puts "beginning"
    doit
    puts "end"
  rescue Exception => e
    puts "ERROR: #{e.message}"
  ensure
    puts "Have a nice day"
  end
  EventMachine.stop
end

From looking at the main block, you have the call to doit wrapped in a begin/rescue, so it seems logic that if an error was raised within it, the rescue block would capture it and handle the error. Unfortunately, the error it being raised outside the context of the fiber, so it isn't caught. You are outside the context of the fiber in the time between when Fiber.yield is called and f.resume is called.

Why is this context important? Because when we built applications, we want them to be resilient. For instance, if a web request encounters an error, we want it caught, reported either to a log or a service, a response returned to the user, and then it be ready to handle the next request. However outside the context of the fiber, the error handling we've built in won't get called and the application can exit. Even if the application is monitored with god, existing connections will get closed out and connections won't be served while it is restarting, and the valuable information within the error is lost.

Once the fiber is resumed, the error handling functions as expected.

require 'eventmachine'
require 'em-synchrony'
require 'fiber'

def doit
  f = Fiber.current
  EventMachine.add_timer(1) { f.resume }
  Fiber.yield
end

EM.synchrony do
  begin
    puts "beginning"
    doit
    raise 'uhh ohh'
  rescue Exception => e
    puts "ERROR: #{e.message}"
  ensure
    puts "Have a nice day"
  end
  EventMachine.stop
end

In this case, the exception will be caught and handled, and you will make it to the "Have a nice day" message.

In order to handle the error, you can add some exception handling within the main context, or the root fiber.

begin
  EM.synchrony do
    begin
      puts "beginning"
      doit
      puts "end"
    rescue Exception => e
      puts "ERROR: #{e.message}"
    ensure
      puts "Have a nice day"
    end
    EventMachine.stop
  end
rescue Exception
  puts "We caught something outside a fiber!"
end

However, I haven't yet figured out how to have the same thing done in the context of a web app. So far, I've just ensured I have proper error handling in the callbacks before the fiber is resumed. Trying to put a begin/rescue block in the config.ru doesn't seem to have any effect on rescuing an error.

The moral of the story is when using fibers, be conscious of error handling outside the context of the fiber. Unhandled exceptions there can cause the entire application to exit. Be aware you have proper error handling when doing something that yields a fiber, and whether they fiber-enabled libraries you are using have error handling outside the context as well.

MySQL Read/Write Splitting in JRuby

2011-05-16T00:00:00-07:00

I was rather surprised about the lack of information out there about how to do read/write splitting with MySQL and Rails on JRuby. This is a pretty key part of our infrastructure, and has been a major point of our attention when performing large platform upgrades.

Having recently gone through one such upgrade and subsequent breaking of our read/write splitting, I had to dive into the code that was doing it more and get a better understanding of how it works.

With JRuby, there are two things to be aware of when trying to enable read/write splitting:

Does it route the queries to the right box?
Does it do it without sucking all available connections up on MySQL?

The second one sounds weird, however we have actually seen it and had it cause major issues. I am not totally sure what to attribute it to, but somewhere between ActiveRecord, ActiveRecord-JDBC, and the MySQL JDBC driver, there is some weird connection handling going on where a connection will still be established, however viewed as dead and a new connection will be opened up. Eventually, it just sucks up all available connections to MySQL.

This is a pretty big problem, but can work around it. Most of the existing information out there about how to do read/write splitting with ActiveRecord doesn't always apply cleanly to JRuby. In particular with one, we ran into the connection-sucking issue while in the benchmarking process.

Fortunately, there isn't a whole lot that needs to be done to enable splitting. The MySQL JDBC driver already has support for it with its ReplicationDriver, you just need to enable it when using Rails. Then you can leverage a plugin which sets a "read_only" property on the raw connection when it recognizes a 'SELECT' query. When read_only is flagged, the driver knows it can send the query to servers other than the master.

Steps to configure read/write splitting:

1) Get activerecord-jdbcmysql-adapter

If you're going to use MySQL on JRuby with ActiveRecord, this is rather a given. We're currently using v1.1.1 and happy. All the JDBC adapters are basically rolled up in one repo which produces multiple gems.

2) Get my version of the active-record-jdbc-mysql-master-slave plugin

There is a plugin that takes care of setting the read_only property on the connection for 'select' queries. The original version of the plugin was originally written for Rails 2.2 and hasn't been updated in 2 years. We originally used it on Rails 2.2.2 and had a lot of success with it.

It doesn't work on Rails 2.3 through, primarily because of the load order changing. It previously worked by alias_methoding the initializer on JdbcAdapter, and in its initialize, would alias_method the _execute method if using mysql. However, load order changed in Rails 2.3 and the database connection is initialized before the plugin is loaded, causing its own version of initialize to not run. I've updated it to use a slightly different method to ensure it happens regardless of load order.

To install the plugin, can just use ./script/plugin:

$ ./script/plugin install git://github.com/krobertson/active-record-jdbc-mysql-master-slave.git

3) Ensure active connections are reset in ActiveRecord

You can still experience some issues with stray connections. The best way we have found to combat this is through an after filter on controllers that does some cleanup. Add this portion to your ApplicationController:

after_filter  :clear_database_connections

def clear_database_connections
  ActiveRecord::Base.clear_active_connections!
end

By calling ActiveRecord::Base.clear_active_connections!, it ensures connections are reset. It is best to try and ensure it is the last after_filter defined.

4) Configure your database.yml

The key in the database.yml is to set up activerecord-jdbc to use MySQL's replication driver and to configure the "url" for it.

production:
  adapter:  jdbcmysql
  username: myuser
  password: secret
<% if $0 =~ /(rake|irb)/ -%>
  host:     masterdb
  port:     3306
  database: myapp
<% else -%>
  driver:   com.mysql.jdbc.ReplicationDriver
  url:      jdbc:mysql://masterdb:3306,slavedb:3306/myapp?roundRobinLoadBalance=false&autoReconnect=true&failOverReadOnly=true
<% end -%>

You may be looking at it and thinking I've got erb in my yaml... and yes. Rails lets you mix the two, and the reason in this case is because we opt to disable the read/write splitting in the Rails console (irb) and when running rake. Two very important reasons to. First, in console, we've had it suck up all connections just when doing simple debugging. Not fun. The normal adapter doesn't do it. And second, we want it off in rake for when we deploy and it does db:migrate. We don't want to allow replication delay in there, such as when querying the schema_migrations table.

For the read/write splitting part, the bulk of the settings are in the "url". The first host specified is the master, and any others are slaves. It is supposed to support multiple slaves and have the option to load balance between them, though we found in practice it never really worked. Instead we direct our web app and background processing to separate slaves. After the hosts comes the database name, and the query string parameters are a number of the driver options.

How the stuff works

At first, looking at the code for the plugin seemed rather confusing. Less than 100 lines of injecting into ActiveRecord did our read/write splitting? But the truth is it is fairly simple, as most of the work is handled by the underlying driver (MySQL's JDBC driver). All we really need to do is populate that read_property on the driver's connection. Shortened up, it does this:

alias_method :_execute_without_master_slave, :_execute
alias_method :_execute, :_execute_with_master_slave

# if we're in auto-commit mode and about to execute a select statement, 
# then set the connection in read-only mode for the duration of
# the query... which will permit the query to be load-balanced
# amongst the slaves by the mysql connector/j ReplicationDriver
def _execute_with_master_slave(sql, name=nil)
  # Need to set the read_only option on the raw connection
  # to tell the underlying driver whether the request can
  # go to slaves.
  cro = raw_connection.connection.read_only
  begin
    raw_connection.connection.read_only = 
      raw_connection.connection.auto_commit &&
      JdbcConnection::select?(sql)

    self._execute_without_master_slave(sql, name)
     
  ensure
    raw_connection.connection.read_only = cro
  end
end

To sum it up, the adapter has an _execute method which does the real execution and is passed in the raw SQL. The plugin injects itself in the middle. First, it captures the current read_only value, because after it executes the given query, it wants to ensure it is set back to that. Next, it sets the read_only property based on whether MySQL itself has auto_commit on and if it is a 'select' query. It then calls the original _execute method, and after it runs, sets read_only back.

The actual plugin has several additional things, like allowing you to run some statements within a block to ensure it goes to the master, and to only work with the MySQL adapter.

Transparency and status

2011-04-29T00:00:00-07:00

One thing that I think can really be highlighted from the recent AWS issues is the value of a status dashboard for your platform/service.. However, simply having one isn't quite enough. Plenty of services already do. What really matters is how effectively it is used.

Just think why a customer would be coming to your status dashboard and what they'd be looking for. If they're coming to it, it is likely because they're having a problem. What they'd be looking for is confirmation that the company is aware of the issue and a rough idea of when it would be fixed.

This sounds quite simple and plainly put, however rarely done. I don't know if its some management/PR intervention of not wanting to sound unstable or unreliable, or if it is just that level of transparency isn't encouraged/practiced at these companies, but simple information often never makes it public, or seems to after the fact.

One our major gripes with Amazon when we were hosted with them was how we'd become aware of an issue, but in some cases it would be hours before an acknowledgment of it went onto the status dashboard. The 1am EBS outage of last week has repeated itself before, and interestingly around 1am several times. It affected Reddit just a month or so ago, and it affected us back in October. One of our other guys was up at 1am, aware immediately of the EBS issue since it was impacting our master database server. It took Amazon two and a half hours to post about it on their status site, even stating it began the same time we were paged.

An additional aspect often missed is the value of the post-mortem. When you have a major issue, customers lose confidence. The post-mortem is a chance to try and gain some of that back. It doesn't need to divulge every single detail and shouldn't look at it as an embarrassment. When I think of a post-mortem, I look for 3 things: what happened, why it happened, and how it won't happen again. It can very effective at reassuring users if you can clearly show that you understand what went wrong and have a solid plan to ensure it won't happen again. Glossing over the chance by simply saying "we've fixed it" (without saying what it is, in some fashion) can simply detract users even more.

Example of a post-mortem done wrong? Look at Tumblr's downtime in December or their security leak in March. Look at the majority of AWS issues. Typically, they don't do them. They only do them when they majorly screw up. And when they do, there is an air of "architect for us" attitude.

Here are a few quick things that would make any status dashboard serve its purpose well.

Update early, update often.
Don't downplay the severity.
Give ETAs on resolutions.
Be very generous with ETAs (go worst case).
Choose a good layout! Look at the image to the side for Amazon's site during their issue.

Above all though, look at the status site from the perspective of their users. Think about why they would come there and what information they're looking for.

Talk Cloudy To Me

2011-04-27T00:00:00-07:00

I will be talking at the Silicon Valley Cloud Computing Meetup this Saturday, April 30th. If you are interested in the topic and in the area, please come out! So far there are about 500+ people plus registered to view panels discussing a wide variety of cloud-related topics throughout the day.

I will be speaking on the panel titled "What are public clouds still missing?" Will be talking about what areas the public clouds still fall short in and where the road ahead will hopefully take us.

Other speakers who will be there include notable people from VMware, including some of the CloudFoundry engineers, Netflix, Cisco, Microsoft, and Rackspace.

Definitely come and check it out! Saturday starting at 1pm, hosted at the Microsoft Mountain View offices.

Amazon and what it means to you

2011-04-22T00:00:00-07:00

Unless you were under a rock, you probably heard about the major issues in Amazon's US-East region yesterday. Needless to say, a lot of businesses had a rough day yesterday.

One of the most interesting aspects of yesterday was realizing how big Amazon's reach is. Numerous well known startups and businesses were down, but beyond that, services not even on Amazon themselves were affected. We moved off EC2 back in December, however we integrate with APIs from several other services, including some that are running on EC2. We still felt the impact and were closely watching Amazon tracking status.

The event lead to some interesting discussions though.

There has been a lot of blame going back and forth, with some leaning on the side of "you should have planned for it". While it can be particularly harsh, in that most best practices for Amazon stressed leveraging multiple availability zones, while yesterday, it was nearly an entire region that was out. There is an element of truth in the argument... have a plan.

Typically that is known as a Disaster Recovery plan. They easy to be overlooked until you realize you need one. They're always one of those "we should do that someday" or maybe even something a company doesn't think its big enough to need.

I am no expert on disaster recovery (and there are plenty of consultants out there to help build a professional one), but there are some pretty basic things you can look at to at least have a plan rather than no plan. Some services recovered pretty well yesterday, albeit with partial functionality, but they got something back up. Others realized they were at the mercy of another provider and were stuck until then.

There are some simple questions you can begin to answer for yourself for a simple plan:

What data is important? Your service runs on code, but it is powered by data. What data is it dependent on? Image assets, file uploads, databases?

Where does my data live, and how can I have it in multiple places? Take regular database backups and store them off site. Keep a backup of assets. It is important for them to be in more than one location since if that location disappears, you are SOL.

How much data loss is acceptable? If stuff goes down, you may lose some data. If you want to limit the loss, you need to keep fresher copies of data. More frequent backups, perhaps a live DB slave hosted somewhere else. You need to either accept that some data loss is acceptable, or architect so that you won't have any data loss.

How fast can you provision servers and where? Geo distribution all the time can be expense and not necessary (unless you deem it is), but being able to have somewhere to quickly set up emergency servers is a major plus. If you use AWS East, then maybe use AWS West for DR. Make sure your account limits give you enough capacity, and have scripts and basic stuff in place and ready to be used. We leverage Chef for all our systems, so we can quickly launch some new servers and get right to work configuring them pretty quickly.

What is your order of priorities? A service is often comprised of many smaller components. Know which ones are the highest priority. Know which functionality without your app is most important. You'll be racing against the clock and have only so many people. This lets them focus their attention to help get core functionality up as quickly as possible.

Do a fire drill. This is something I think we'll be doing pretty soon. Out of the blue one day, do a mock fire drill. You are down, your datacenter disappeared. Get the service back up in an alternative place. How long does it take you? What were the pain points? Beyond simply having a plan, you need to know the plan will work.

And lastly, know your application. Know its requirements, constraints, and understand its bare minimums, most important pieces, etc.

Power through environments

2011-04-14T00:00:00-07:00

At Involver, we have a pretty rich deployment environment that I think isn't very common. It is normal to hear about places having a separate staging environment from the production environment, and sometimes having a staging and test environment. We actually go a step further and have a total of 8 non-production environments. They all serve different purposes, but are all heavily used.

Our environments are broken up like this:

Production: duh, live stuff
Staging: Used for QA, close to production, no experimental stuff, pretty good data set and history in the DB.
Demo: Used for client-facing stuff that is under development. Somewhat experimental, but expected to not be breaking or of a major impact.
Perf: This environment is regularly repurposed and is used for benchmarking, testing infrastructure changes, and other large scale changes (Rails upgrades, DB upgrades, core app perf testing).
Test1-5: These guys are pretty special.

The test environments are something special and are meant to be highly flexible. They are used to deploy separate git branches to an environment that has all the different elements of our application fully orchestrated. They are useful for testing volatile code, experimental code, or for performing QA on features or feature branches before the code gets merged into master.

The deployment to these environments works by specifying the branch on deploy, and it will create the MySQL/Mongo databases, setup the DB schema, bootstrap the DB with some initial data, and then configure all the configuration files and even config files for other applications that it interacts with. Each branch on each environment gets its own database. And between deploys, the data for your branch is fully preserved.

So say you deploy "dev-fix-booboo" to test3 for the first time. It sets you up with a fresh dataset and are good to go. You do your thing, then later on, someone deploys another branch. They get their own DB, not mucking up the stuff you previously had done, and when you deploy there again, you are right back where you left off.

Since they are specific to the environment too, we often times assign a long-running feature branch to an environment, so that the data used by QA sticks around more, and smaller test branches can use them here and there.

The full power of these comes form:

Flexibility. I have a branch I want to deploy, but not much other environments. Sweet.
Some element of persistence. I can use it for a few days, come back, and be right where I left off. No risk of someone polluting my stuff with bad data.
QA can actually deploy to them and pick which is available. For non-feature branch testing, this gives them great flexibility.

Of course, there are still some pitfalls:

They don't mirror the full production setup. They're usually one box and not spec'd for any kind of load. They run good, but not going to handle lots of people. Can't perf test here.
There is the data persistence between switching branches, but often no rich DB history. It is harder to test large datasets of legacy type setups.
Can't test infrastructure changes. If deploys don't switch out gems or system configs, so harder if you want to test a change like that, you need to lock the environment til done.

All of this makes for a very flexible deployment process. It allows devs to easily get code off of their machines and in a more controlled environment. It allows us to have high confidence in code before it makes it to master and often even before the feature branch. It also allows QA to do their testing without potentially deploying code they reject to staging or demo, where it might be customer facing or impact other development.

Big props for it go to @mikewadhera, who initially wrote the deployment process for it. Since then, we (infra/ops team) have worked on making the environments single-system, which allow us to have more of them, as well as iterate on making them more streamlined with developer and QA feedback.

This kind of builds up to something I'll be talking about soon too. Since we have a total of 9 different environments, this can lead to a very interesting chef workflow.

Serving Static Content Via POST From Nginx

2011-04-12T00:00:00-07:00

Recently, a change was made on Facebook's end where their <fb:iframe> FBML tag started doing POST request to get content. Their documentation seemed to indicate their "Post for Canvas" change wouldn't affect the <fb:iframe> tag, however our error logs spoke quite to the contrary. We saw an influx in errors in places where we used <fb:iframe> to pull in static content. Nginx would simply return a 405 error for "Method Not Allowed". Nginx can't serve static content on a POST request.

Some quick googling seems to reveal some quick answers, but the problem is none of them worked.

The most popular, this thread, talks about a work around by manipulating the error page for nginx. First, there is some mixed syntax depending on the nginx version. Some references seem to be to old 2007 syntax which doesn't work on the nginx 0.7 build we use. But they all indicated that this should have worked:

error_page 405 =200 @405;
location @405 {
  root ...;
}

This basically tells nginx to change the response code to 200 for 405 messages, and to use the @405 location entry to handle it. Setting the root it is supposed to pull the requested document. It didn't work though. At best, I got it to return a 200 code, but no response body. Essentially, it was still doing a POST request and it still didn't like it.

I could however, get it to be served by proxying it to our Rails application, if I enabled serving static content from Rails. But that was less than ideal.

But then, I was scanning through nginx's proxy documentation and found what seemed like an ideal solution. I could configure nginx to serve static content on another port, have the main server proxy it to the other port, and when proxying, change it into a GET request:

upstream static_backend {
  server localhost:89;
}

server {
  ...
  error_page 405 =200 @405;
  location @405 {
    root ...;
    proxy_method GET;
    proxy_pass http://static_backend;
  }
}

# POST STATIC CONTENT
server {
  listen 89;
  server_name _;
  root ...;
}