In Valid Logic

Jekyll on PaaS.io with Cloud Foundry

Fri Jan 06 00:00:00 -0800 2012

Recently I moved my blog over to the service I am currently working on building, PaaS.io.

Running Jekyll on PaaS.io isn't all that different from running it on other services, though I had a few other goals in mind. The way I wanted it set up was:

Stop using rack-jekyll. Its a nice gem, however it is locked to an older version of Jekyll. And the current gem is on an even more outdated one. Currently, Cloud Foundry doesn't support pulling bundler sources from git too.
Have a public folder for static content like CSS and images.
Have the _site folder for generated content
Don't have it copy the Gemfile and the config.ru into the _site folder (annoys me)
Redirect www.invalidlogic.com to invalidlogic.com
Low foot print

First, the Gemfile:

source :rubygems

gem 'rack-contrib', :require => 'rack/contrib/try_static'
gem 'rack-redirect'
gem 'thin'

group :development do
  gem 'jekyll'
  gem 'RedCloth'
  gem 'rdiscount'
end

The main gems being used are thin, rack-contrib (for TryStatic, note on that later), and rack-redirect (for www redirection). I also include some of the gems I use for Jekyll in the development group. That way they are available locally but not loaded when I deploy (lower footprint... and yes, it is minor).

Now, the config.ru:

require 'rubygems'
require 'bundler'
Bundler.require

use Rack::EY::Solo::DomainRedirect

use Rack::TryStatic,
    :root => "_site",
    :urls => %w[/],
    :try => ['.html', 'index.html', '/index.html']

use Rack::Static,
    :root => "public",
    :urls => %w[/]

run lambda { [404, {'Content-Type' => 'text/html'}, ['Not Found']]}

There are 4 rack components here. First, Rack::EY::Solo::DomainRedirect is the rack-redirect gem and handles the www redirection. Next, is Rack::TryStatic. It is used to access files from the _site generated content directory. It gives a couple different :try values for different ways to find the intended file. Then is the Rack::Static which gets static content from the public directory. No need to try different combinations. And last is a generic lambda that will return 404.

Next, want to avoid duplication. With things as they are, when I run jekyll it will copy the public and other items into the _site folder duplicating it. To resolve that, in our _config.yml, can add an exclude line:

exclude: [ 'public', 'Gemfile', 'Gemfile.lock', 'config.ru' ]

And with that, we are set! All of our goals are met. Commit and push to deploy! Currently I have Cloud Foundry set up with a Rack framework defined (will be sending a pull request with it soon) and also have my blog set to use Ruby 1.9.3 as well.

Soon I'll be providing some more details on PaaS.io, so stay tuned and click over to it and sign up to get access to the beta.

iPhone vs Android is the new Mac vs PC

Tue Oct 18 00:00:00 -0700 2011

After spending 18 months with Android, I am now back to an iPhone and likely here to stay. While Android isn't necessarily bad, it more boils down to iPhone simply being better. After a while, I started looking at iPhone vs Android as largely a repeat of the Mac vs PC comparisons.

Mac vs PC

Just look at it. Apple is doing what they've always done. They control the hardware and the software. They have a solid, unified experience. You can pick up any iPhone, old or new, and still be at home.

Android is repeating the history of PCs. Google makes the OS, as Microsoft did with Windows. Then OEMs offer a wide variety of different hardware, with their own shitty customizations layered on top of the OS. Even worse, you have the carriers layering on their customizations and restrictions.

You can go from one Android phone to another and have it be completely foreign. Even baseline apps can have different names and a completely different look and feel, such between the "Email" vs "Mail" of vanilla Android and HTC Android.

Shelf Life

With Android, the phones have a very short shelf life. I bought a Thunderbolt from Verizon back in April, just 6 months ago. About a month after I bought it, it was no longer the hot model they were pimping. In fact, there have been 2-3 phones to come out since then that became "king of the mountain."

With iPhone, they release a new phone about once a year, and that one stays the current phone. Older phones are still pretty well supported. The iPhone 3GS is over 2 years old, still got updated to iOS 5, and likely will until iOS 6. My original Motorola Droid that is nearing 2 years old is pretty much forgotten already.

Updates with iPhone? Available right away to everyone. AT&T or Verizon, you get the update when Apple makes it generally available.

Updates with Android? Have fun. Google has to release it, your OEM has to customize it, then the carrier gets to tweak it, and decide when to finally roll it out. Google released Gingerbread back in December 2010, nearly 10 months ago. Verizon just started rolling out the update last month, then pulled it. Even then, they decide when you can upgrade with a phased roll out. And since it was pulled, looks like they seem to skip over adequate testing.

Most Android users who want recent releases end up rooting their phones and use unofficial ROMs put together by an informal group of people. Have an issue with your phone? Limited options.

Marketing

I definitely agreed with my friend Joe in his post about the Droid Bionic. Android phones are being pitched like PCs. They give a bunch of technical specs that are meaningless to consumers. It echos the "Golden Circle" idea by Simon Sinek. Apple is still doing what they do best. Verizon sells like PC manufacturers. Lacking inspiration.

False Sense of Market Share

You've probably seen the headlines of "Android sales outpacing iPhone" or "Android market share to surpass iPhone".

Overall the numbers are comparing apples to oranges. They are comparing the sales figure of a single phone to a whole group of phones. There are many different Android phones, and individually, iPhone is spanking them in sales figures. No single Android phone has a chance of elipsing the iPhone or gaining any meaningful market share, especially with their overall short shelf life. I'd be very interested to see sales figured of individual phones and see how long they really last. All the manufacturers are still struggling to get a sliver of what the iPhone is capable of.

The numbers may hold some weight for developers because it represents the size of your audience. But at the same time, Android is many phones vs iPhones entire existence is now just 5 phones. Less "your app doesn't work on my Verizon Droid Mumbojumbo" and you don't have a Droid Mumbojumbo to test on.

Falseness of Open

Many people tout Android as being open, however the actions of Google with Honeycomb's source code seem to be heading in the direction of more closed.

While Google said part of their goal was to try and unify the platform more, as Honeycomb was intended for tablets and not handsets, the ability to control a platform is difficult while also keeping it "open." I think Google is right in closing it, since in order to further the platform, they will need to have some control in order to maintain a consistent direction.

However, the "openness" usually just comes from developers. What do most of then define the "openness" as? Being able to write apps and put them on their phone without paying $99. They tout the source being open, but the truth is that isn't what they really care about. Very few Android developers likely dive into the OS code, they just want to install their own apps for free.

Personally, if I prefer one platform over the other, a $99 fee to build apps isn't going to a stopper for me. But I also don't mind paying for the tools I use in my craft if they are worth it.

Working

Most average people care more about how well the phone works. iPhone simply works better. Since software and hardware are more married, the experience is more consistent. In my own experience, apps crash less on iPhone. The phone lags less. Scrolling and browsing is more graceful. My wife is still on her original Droid for another month, and every day I see her struggle with the phone.

iPhone has better customer satisfaction ratings than Android. While the reasons aren't mentioned, I wouldn't be surprised that part of it stems from "it just works".

Apple has a strong emphasis on usability. Google and the OEMs aren't as much so. This particularly stuck me with a post I read about a guy talking to a girl who had an Android phone and an iPod Touch. In particuar, "nothing happened when I plugged it into my computer." To the average user, the simple integration is important. They don't know why the phone shows up as a "Mass Storage Device" when they plug it in.

Future

The future for Android will probable improve. The OS is maturing and hardware getting better, however the ecosystem of manufacturers and carriers will likely stay the same. The reality is that Apple is doing what Apple has always done and they've gotten really good at it.

Cooking with Chef slides

Mon Jul 25 00:00:00 -0700 2011

A little late, but I have posted my slides from my talk at the East Bay Ruby Meetup in June.

Cooking with Chef

Check them out and feel free to ping me if there are any questions. At the meetup, was also talking about doing a bit of a blog series on getting started with Chef and posting some of the scripts and baseline setup I have used before. Hope to start forming some simple getting started resources.

New Opportunities

Tue Jul 19 00:00:00 -0700 2011

Finally posting about it a little bit late, but last week was my first week in a new position with a new company. I truly enjoyed my stay at Involver, but saw an opportunity pop up and decided to take it.

This new opportunity is to be one of the first in-house Ruby developers at Demandbase. Demandbase specializes in a B2B analytics API for reporting and logging information about visitors hitting your business's website.

That all sounds fancy, but the gist is I'll be helping to grow and form the team, scale up and expand the platform, and able to leverage both skillsets as a developer and in operations. They've been working with the awesome guys from Highgroove, utilizing Chef in a continuous deployment setup, doing geographic load balancing, and have a pretty agile and responsive process setup.

I had an amazing time at Involver and grew so much personally and professionally. I was a part of some major projects and instrumented a lot of change. Our Operations team was top notch and put out of a lot of stuff considering it was only 3 people.

Going forward, have some goodies I'll hopefully be working on and open sourcing. There are a lot of interesting things in the works and hope to post about them over time.

And by the way, Demandbase is hiring. Hit me up @krobertson or shoot me an email.

Announcing Gemstache

Thu Jun 30 00:00:00 -0700 2011

Pleased to announce Gemstache, a new service to enable teams and companies to build and distribute their own private gems with security and ease of use in mind.

Gems are an incredibly simple way to package up code and distribute it, making it available to multiple code bases or just a great way to encapsulate functionality. However, the way gems are traditionally distributed doesn't lend itself well to securing the gems and controlling access. The default "gem install" command you run doesn't provide any support for authentication against the gem source, so most sources are open to the public.

Gemstache works by giving you your own private gem source you can use. In order to download any gems from it, you must have the accompanied "gemstache" gem installed locally or on your servers. When talking to your private gem source, it interlaces to add authentication to the request as well as ensuring it is happening over HTTPS. With this, your gems are only downloaded over SSL and any interaction requires authentication.

To make managing your gems easier, the gem adds the ability to easily upload a gem to your private gem source and make it available. From your command line, just run "gem stache [your gem file]" and it is uploaded and ready for use.

Within the service, you can give users varying roles to define whether they can just download gems, are able to publish new gems, and are able to yank/delete gems.

Gemstache was born out of a need we had been facing at Involver, where some of our own internal code is enclosed in gems and used between a few of our codebases, as well as we have some gems we have forked to either add customizations or our own fixes. Traditional methods left them exposed and also more painstaking to release.

We're just about ready to launch a public beta. Visit gemstache.com and sign up to be notified.

Beware of error handling with fibers in an evented context

Thu May 26 00:00:00 -0700 2011

One of the more interesting developments in Ruby lately has been the interesting things being done with Ruby 1.9.2, Fibers, and EventMachine. Whether you look at it being applied to existing libraries with rack/fiber_pool, or Goliath, it packs a punch and proving interesting for high performance evented applications.

Recently when reworking one of our backend applications to use rack/fiber_pool with EventMachine, I ran into a gotcha when it came to exception handling. Asynchronous applications makes heavy use of callbacks, where the context of when an operation is completed is not inline with when it began. Fibers work to make it easier to write asynchronous code look like synchronous, where you make a call, and it returns a result.

The problem is that this can play a trick on your eye. You think you are executing something in line, but the context it is operating within jumps without you being necessarily aware of it. Take the following example:

require 'eventmachine'
require 'em-synchrony'
require 'fiber'

def doit
  f = Fiber.current
  EventMachine.add_timer(1) { raise 'uhh ohh' }
  Fiber.yield
end

EM.synchrony do
  begin
    puts "beginning"
    doit
    puts "end"
  rescue Exception => e
    puts "ERROR: #{e.message}"
  ensure
    puts "Have a nice day"
  end
  EventMachine.stop
end

From looking at the main block, you have the call to doit wrapped in a begin/rescue, so it seems logic that if an error was raised within it, the rescue block would capture it and handle the error. Unfortunately, the error it being raised outside the context of the fiber, so it isn't caught. You are outside the context of the fiber in the time between when Fiber.yield is called and f.resume is called.

Why is this context important? Because when we built applications, we want them to be resilient. For instance, if a web request encounters an error, we want it caught, reported either to a log or a service, a response returned to the user, and then it be ready to handle the next request. However outside the context of the fiber, the error handling we've built in won't get called and the application can exit. Even if the application is monitored with god, existing connections will get closed out and connections won't be served while it is restarting, and the valuable information within the error is lost.

Once the fiber is resumed, the error handling functions as expected.

require 'eventmachine'
require 'em-synchrony'
require 'fiber'

def doit
  f = Fiber.current
  EventMachine.add_timer(1) { f.resume }
  Fiber.yield
end

EM.synchrony do
  begin
    puts "beginning"
    doit
    raise 'uhh ohh'
  rescue Exception => e
    puts "ERROR: #{e.message}"
  ensure
    puts "Have a nice day"
  end
  EventMachine.stop
end

In this case, the exception will be caught and handled, and you will make it to the "Have a nice day" message.

In order to handle the error, you can add some exception handling within the main context, or the root fiber.

begin
  EM.synchrony do
    begin
      puts "beginning"
      doit
      puts "end"
    rescue Exception => e
      puts "ERROR: #{e.message}"
    ensure
      puts "Have a nice day"
    end
    EventMachine.stop
  end
rescue Exception
  puts "We caught something outside a fiber!"
end

However, I haven't yet figured out how to have the same thing done in the context of a web app. So far, I've just ensured I have proper error handling in the callbacks before the fiber is resumed. Trying to put a begin/rescue block in the config.ru doesn't seem to have any effect on rescuing an error.

The moral of the story is when using fibers, be conscious of error handling outside the context of the fiber. Unhandled exceptions there can cause the entire application to exit. Be aware you have proper error handling when doing something that yields a fiber, and whether they fiber-enabled libraries you are using have error handling outside the context as well.

MySQL Read/Write Splitting in JRuby

Mon May 16 00:00:00 -0700 2011

I was rather surprised about the lack of information out there about how to do read/write splitting with MySQL and Rails on JRuby. This is a pretty key part of our infrastructure, and has been a major point of our attention when performing large platform upgrades.

Having recently gone through one such upgrade and subsequent breaking of our read/write splitting, I had to dive into the code that was doing it more and get a better understanding of how it works.

With JRuby, there are two things to be aware of when trying to enable read/write splitting:

Does it route the queries to the right box?
Does it do it without sucking all available connections up on MySQL?

The second one sounds weird, however we have actually seen it and had it cause major issues. I am not totally sure what to attribute it to, but somewhere between ActiveRecord, ActiveRecord-JDBC, and the MySQL JDBC driver, there is some weird connection handling going on where a connection will still be established, however viewed as dead and a new connection will be opened up. Eventually, it just sucks up all available connections to MySQL.

This is a pretty big problem, but can work around it. Most of the existing information out there about how to do read/write splitting with ActiveRecord doesn't always apply cleanly to JRuby. In particular with one, we ran into the connection-sucking issue while in the benchmarking process.

Fortunately, there isn't a whole lot that needs to be done to enable splitting. The MySQL JDBC driver already has support for it with its ReplicationDriver, you just need to enable it when using Rails. Then you can leverage a plugin which sets a "read_only" property on the raw connection when it recognizes a 'SELECT' query. When read_only is flagged, the driver knows it can send the query to servers other than the master.

Steps to configure read/write splitting:

1) Get activerecord-jdbcmysql-adapter

If you're going to use MySQL on JRuby with ActiveRecord, this is rather a given. We're currently using v1.1.1 and happy. All the JDBC adapters are basically rolled up in one repo which produces multiple gems.

2) Get my version of the active-record-jdbc-mysql-master-slave plugin

There is a plugin that takes care of setting the read_only property on the connection for 'select' queries. The original version of the plugin was originally written for Rails 2.2 and hasn't been updated in 2 years. We originally used it on Rails 2.2.2 and had a lot of success with it.

It doesn't work on Rails 2.3 through, primarily because of the load order changing. It previously worked by alias_methoding the initializer on JdbcAdapter, and in its initialize, would alias_method the _execute method if using mysql. However, load order changed in Rails 2.3 and the database connection is initialized before the plugin is loaded, causing its own version of initialize to not run. I've updated it to use a slightly different method to ensure it happens regardless of load order.

To install the plugin, can just use ./script/plugin:

$ ./script/plugin install git://github.com/krobertson/active-record-jdbc-mysql-master-slave.git

3) Ensure active connections are reset in ActiveRecord

You can still experience some issues with stray connections. The best way we have found to combat this is through an after filter on controllers that does some cleanup. Add this portion to your ApplicationController:

after_filter  :clear_database_connections

def clear_database_connections
  ActiveRecord::Base.clear_active_connections!
end

By calling ActiveRecord::Base.clear_active_connections!, it ensures connections are reset. It is best to try and ensure it is the last after_filter defined.

4) Configure your database.yml

The key in the database.yml is to set up activerecord-jdbc to use MySQL's replication driver and to configure the "url" for it.

production:
  adapter:  jdbcmysql
  username: myuser
  password: secret
<% if $0 =~ /(rake|irb)/ -%>
  host:     masterdb
  port:     3306
  database: myapp
<% else -%>
  driver:   com.mysql.jdbc.ReplicationDriver
  url:      jdbc:mysql://masterdb:3306,slavedb:3306/myapp?roundRobinLoadBalance=false&autoReconnect=true&failOverReadOnly=true
<% end -%>

You may be looking at it and thinking I've got erb in my yaml... and yes. Rails lets you mix the two, and the reason in this case is because we opt to disable the read/write splitting in the Rails console (irb) and when running rake. Two very important reasons to. First, in console, we've had it suck up all connections just when doing simple debugging. Not fun. The normal adapter doesn't do it. And second, we want it off in rake for when we deploy and it does db:migrate. We don't want to allow replication delay in there, such as when querying the schema_migrations table.

For the read/write splitting part, the bulk of the settings are in the "url". The first host specified is the master, and any others are slaves. It is supposed to support multiple slaves and have the option to load balance between them, though we found in practice it never really worked. Instead we direct our web app and background processing to separate slaves. After the hosts comes the database name, and the query string parameters are a number of the driver options.

How the stuff works

At first, looking at the code for the plugin seemed rather confusing. Less than 100 lines of injecting into ActiveRecord did our read/write splitting? But the truth is it is fairly simple, as most of the work is handled by the underlying driver (MySQL's JDBC driver). All we really need to do is populate that read_property on the driver's connection. Shortened up, it does this:

alias_method :_execute_without_master_slave, :_execute
alias_method :_execute, :_execute_with_master_slave

# if we're in auto-commit mode and about to execute a select statement, 
# then set the connection in read-only mode for the duration of
# the query... which will permit the query to be load-balanced
# amongst the slaves by the mysql connector/j ReplicationDriver
def _execute_with_master_slave(sql, name=nil)
  # Need to set the read_only option on the raw connection
  # to tell the underlying driver whether the request can
  # go to slaves.
  cro = raw_connection.connection.read_only
  begin
    raw_connection.connection.read_only = 
      raw_connection.connection.auto_commit &&
      JdbcConnection::select?(sql)

    self._execute_without_master_slave(sql, name)
     
  ensure
    raw_connection.connection.read_only = cro
  end
end

To sum it up, the adapter has an _execute method which does the real execution and is passed in the raw SQL. The plugin injects itself in the middle. First, it captures the current read_only value, because after it executes the given query, it wants to ensure it is set back to that. Next, it sets the read_only property based on whether MySQL itself has auto_commit on and if it is a 'select' query. It then calls the original _execute method, and after it runs, sets read_only back.

The actual plugin has several additional things, like allowing you to run some statements within a block to ensure it goes to the master, and to only work with the MySQL adapter.

Transparency and status

Fri Apr 29 00:00:00 -0700 2011

One thing that I think can really be highlighted from the recent AWS issues is the value of a status dashboard for your platform/service.. However, simply having one isn't quite enough. Plenty of services already do. What really matters is how effectively it is used.

Just think why a customer would be coming to your status dashboard and what they'd be looking for. If they're coming to it, it is likely because they're having a problem. What they'd be looking for is confirmation that the company is aware of the issue and a rough idea of when it would be fixed.

This sounds quite simple and plainly put, however rarely done. I don't know if its some management/PR intervention of not wanting to sound unstable or unreliable, or if it is just that level of transparency isn't encouraged/practiced at these companies, but simple information often never makes it public, or seems to after the fact.

One our major gripes with Amazon when we were hosted with them was how we'd become aware of an issue, but in some cases it would be hours before an acknowledgment of it went onto the status dashboard. The 1am EBS outage of last week has repeated itself before, and interestingly around 1am several times. It affected Reddit just a month or so ago, and it affected us back in October. One of our other guys was up at 1am, aware immediately of the EBS issue since it was impacting our master database server. It took Amazon two and a half hours to post about it on their status site, even stating it began the same time we were paged.

An additional aspect often missed is the value of the post-mortem. When you have a major issue, customers lose confidence. The post-mortem is a chance to try and gain some of that back. It doesn't need to divulge every single detail and shouldn't look at it as an embarrassment. When I think of a post-mortem, I look for 3 things: what happened, why it happened, and how it won't happen again. It can very effective at reassuring users if you can clearly show that you understand what went wrong and have a solid plan to ensure it won't happen again. Glossing over the chance by simply saying "we've fixed it" (without saying what it is, in some fashion) can simply detract users even more.

Example of a post-mortem done wrong? Look at Tumblr's downtime in December or their security leak in March. Look at the majority of AWS issues. Typically, they don't do them. They only do them when they majorly screw up. And when they do, there is an air of "architect for us" attitude.

Here are a few quick things that would make any status dashboard serve its purpose well.

Update early, update often.
Don't downplay the severity.
Give ETAs on resolutions.
Be very generous with ETAs (go worst case).
Choose a good layout! Look at the image to the side for Amazon's site during their issue.

Above all though, look at the status site from the perspective of their users. Think about why they would come there and what information they're looking for.

Talk Cloudy To Me

Wed Apr 27 00:00:00 -0700 2011

I will be talking at the Silicon Valley Cloud Computing Meetup this Saturday, April 30th. If you are interested in the topic and in the area, please come out! So far there are about 500+ people plus registered to view panels discussing a wide variety of cloud-related topics throughout the day.

I will be speaking on the panel titled "What are public clouds still missing?" Will be talking about what areas the public clouds still fall short in and where the road ahead will hopefully take us.

Other speakers who will be there include notable people from VMware, including some of the CloudFoundry engineers, Netflix, Cisco, Microsoft, and Rackspace.

Definitely come and check it out! Saturday starting at 1pm, hosted at the Microsoft Mountain View offices.

Amazon and what it means to you

Fri Apr 22 00:00:00 -0700 2011

Unless you were under a rock, you probably heard about the major issues in Amazon's US-East region yesterday. Needless to say, a lot of businesses had a rough day yesterday.

One of the most interesting aspects of yesterday was realizing how big Amazon's reach is. Numerous well known startups and businesses were down, but beyond that, services not even on Amazon themselves were affected. We moved off EC2 back in December, however we integrate with APIs from several other services, including some that are running on EC2. We still felt the impact and were closely watching Amazon tracking status.

The event lead to some interesting discussions though.

There has been a lot of blame going back and forth, with some leaning on the side of "you should have planned for it". While it can be particularly harsh, in that most best practices for Amazon stressed leveraging multiple availability zones, while yesterday, it was nearly an entire region that was out. There is an element of truth in the argument... have a plan.

Typically that is known as a Disaster Recovery plan. They easy to be overlooked until you realize you need one. They're always one of those "we should do that someday" or maybe even something a company doesn't think its big enough to need.

I am no expert on disaster recovery (and there are plenty of consultants out there to help build a professional one), but there are some pretty basic things you can look at to at least have a plan rather than no plan. Some services recovered pretty well yesterday, albeit with partial functionality, but they got something back up. Others realized they were at the mercy of another provider and were stuck until then.

There are some simple questions you can begin to answer for yourself for a simple plan:

What data is important? Your service runs on code, but it is powered by data. What data is it dependent on? Image assets, file uploads, databases?

Where does my data live, and how can I have it in multiple places? Take regular database backups and store them off site. Keep a backup of assets. It is important for them to be in more than one location since if that location disappears, you are SOL.

How much data loss is acceptable? If stuff goes down, you may lose some data. If you want to limit the loss, you need to keep fresher copies of data. More frequent backups, perhaps a live DB slave hosted somewhere else. You need to either accept that some data loss is acceptable, or architect so that you won't have any data loss.

How fast can you provision servers and where? Geo distribution all the time can be expense and not necessary (unless you deem it is), but being able to have somewhere to quickly set up emergency servers is a major plus. If you use AWS East, then maybe use AWS West for DR. Make sure your account limits give you enough capacity, and have scripts and basic stuff in place and ready to be used. We leverage Chef for all our systems, so we can quickly launch some new servers and get right to work configuring them pretty quickly.

What is your order of priorities? A service is often comprised of many smaller components. Know which ones are the highest priority. Know which functionality without your app is most important. You'll be racing against the clock and have only so many people. This lets them focus their attention to help get core functionality up as quickly as possible.

Do a fire drill. This is something I think we'll be doing pretty soon. Out of the blue one day, do a mock fire drill. You are down, your datacenter disappeared. Get the service back up in an alternative place. How long does it take you? What were the pain points? Beyond simply having a plan, you need to know the plan will work.

And lastly, know your application. Know its requirements, constraints, and understand its bare minimums, most important pieces, etc.