In Valid Logic

Jekyll on PaaS.io with Cloud Foundry

2012-01-06T00:00:00-08:00

Recently I moved my blog over to the service I am currently working on building, PaaS.io.

Running Jekyll on PaaS.io isn't all that different from running it on other services, though I had a few other goals in mind. The way I wanted it set up was:

Stop using rack-jekyll. Its a nice gem, however it is locked to an older version of Jekyll. And the current gem is on an even more outdated one. Currently, Cloud Foundry doesn't support pulling bundler sources from git too.
Have a public folder for static content like CSS and images.
Have the _site folder for generated content
Don't have it copy the Gemfile and the config.ru into the _site folder (annoys me)
Redirect www.invalidlogic.com to invalidlogic.com
Low foot print

First, the Gemfile:

source :rubygems

gem 'rack-contrib', :require => 'rack/contrib/try_static'
gem 'rack-redirect'
gem 'thin'

group :development do
  gem 'jekyll'
  gem 'RedCloth'
  gem 'rdiscount'
end

The main gems being used are thin, rack-contrib (for TryStatic, note on that later), and rack-redirect (for www redirection). I also include some of the gems I use for Jekyll in the development group. That way they are available locally but not loaded when I deploy (lower footprint... and yes, it is minor).

Now, the config.ru:

require 'rubygems'
require 'bundler'
Bundler.require

use Rack::EY::Solo::DomainRedirect

use Rack::TryStatic,
    :root => "_site",
    :urls => %w[/],
    :try => ['.html', 'index.html', '/index.html']

use Rack::Static,
    :root => "public",
    :urls => %w[/]

run lambda { [404, {'Content-Type' => 'text/html'}, ['Not Found']]}

There are 4 rack components here. First, Rack::EY::Solo::DomainRedirect is the rack-redirect gem and handles the www redirection. Next, is Rack::TryStatic. It is used to access files from the _site generated content directory. It gives a couple different :try values for different ways to find the intended file. Then is the Rack::Static which gets static content from the public directory. No need to try different combinations. And last is a generic lambda that will return 404.

Next, want to avoid duplication. With things as they are, when I run jekyll it will copy the public and other items into the _site folder duplicating it. To resolve that, in our _config.yml, can add an exclude line:

exclude: [ 'public', 'Gemfile', 'Gemfile.lock', 'config.ru' ]

And with that, we are set! All of our goals are met. Commit and push to deploy! Currently I have Cloud Foundry set up with a Rack framework defined (will be sending a pull request with it soon) and also have my blog set to use Ruby 1.9.3 as well.

Soon I'll be providing some more details on PaaS.io, so stay tuned and click over to it and sign up to get access to the beta.

iPhone vs Android is the new Mac vs PC

2011-10-18T00:00:00-07:00

After spending 18 months with Android, I am now back to an iPhone and likely here to stay. While Android isn't necessarily bad, it more boils down to iPhone simply being better. After a while, I started looking at iPhone vs Android as largely a repeat of the Mac vs PC comparisons.

Mac vs PC

Just look at it. Apple is doing what they've always done. They control the hardware and the software. They have a solid, unified experience. You can pick up any iPhone, old or new, and still be at home.

Android is repeating the history of PCs. Google makes the OS, as Microsoft did with Windows. Then OEMs offer a wide variety of different hardware, with their own shitty customizations layered on top of the OS. Even worse, you have the carriers layering on their customizations and restrictions.

You can go from one Android phone to another and have it be completely foreign. Even baseline apps can have different names and a completely different look and feel, such between the "Email" vs "Mail" of vanilla Android and HTC Android.

Shelf Life

With Android, the phones have a very short shelf life. I bought a Thunderbolt from Verizon back in April, just 6 months ago. About a month after I bought it, it was no longer the hot model they were pimping. In fact, there have been 2-3 phones to come out since then that became "king of the mountain."

With iPhone, they release a new phone about once a year, and that one stays the current phone. Older phones are still pretty well supported. The iPhone 3GS is over 2 years old, still got updated to iOS 5, and likely will until iOS 6. My original Motorola Droid that is nearing 2 years old is pretty much forgotten already.

Updates with iPhone? Available right away to everyone. AT&T or Verizon, you get the update when Apple makes it generally available.

Updates with Android? Have fun. Google has to release it, your OEM has to customize it, then the carrier gets to tweak it, and decide when to finally roll it out. Google released Gingerbread back in December 2010, nearly 10 months ago. Verizon just started rolling out the update last month, then pulled it. Even then, they decide when you can upgrade with a phased roll out. And since it was pulled, looks like they seem to skip over adequate testing.

Most Android users who want recent releases end up rooting their phones and use unofficial ROMs put together by an informal group of people. Have an issue with your phone? Limited options.

Marketing

I definitely agreed with my friend Joe in his post about the Droid Bionic. Android phones are being pitched like PCs. They give a bunch of technical specs that are meaningless to consumers. It echos the "Golden Circle" idea by Simon Sinek. Apple is still doing what they do best. Verizon sells like PC manufacturers. Lacking inspiration.

False Sense of Market Share

You've probably seen the headlines of "Android sales outpacing iPhone" or "Android market share to surpass iPhone".

Overall the numbers are comparing apples to oranges. They are comparing the sales figure of a single phone to a whole group of phones. There are many different Android phones, and individually, iPhone is spanking them in sales figures. No single Android phone has a chance of elipsing the iPhone or gaining any meaningful market share, especially with their overall short shelf life. I'd be very interested to see sales figured of individual phones and see how long they really last. All the manufacturers are still struggling to get a sliver of what the iPhone is capable of.

The numbers may hold some weight for developers because it represents the size of your audience. But at the same time, Android is many phones vs iPhones entire existence is now just 5 phones. Less "your app doesn't work on my Verizon Droid Mumbojumbo" and you don't have a Droid Mumbojumbo to test on.

Falseness of Open

Many people tout Android as being open, however the actions of Google with Honeycomb's source code seem to be heading in the direction of more closed.

While Google said part of their goal was to try and unify the platform more, as Honeycomb was intended for tablets and not handsets, the ability to control a platform is difficult while also keeping it "open." I think Google is right in closing it, since in order to further the platform, they will need to have some control in order to maintain a consistent direction.

However, the "openness" usually just comes from developers. What do most of then define the "openness" as? Being able to write apps and put them on their phone without paying $99. They tout the source being open, but the truth is that isn't what they really care about. Very few Android developers likely dive into the OS code, they just want to install their own apps for free.

Personally, if I prefer one platform over the other, a $99 fee to build apps isn't going to a stopper for me. But I also don't mind paying for the tools I use in my craft if they are worth it.

Working

Most average people care more about how well the phone works. iPhone simply works better. Since software and hardware are more married, the experience is more consistent. In my own experience, apps crash less on iPhone. The phone lags less. Scrolling and browsing is more graceful. My wife is still on her original Droid for another month, and every day I see her struggle with the phone.

iPhone has better customer satisfaction ratings than Android. While the reasons aren't mentioned, I wouldn't be surprised that part of it stems from "it just works".

Apple has a strong emphasis on usability. Google and the OEMs aren't as much so. This particularly stuck me with a post I read about a guy talking to a girl who had an Android phone and an iPod Touch. In particuar, "nothing happened when I plugged it into my computer." To the average user, the simple integration is important. They don't know why the phone shows up as a "Mass Storage Device" when they plug it in.

Future

The future for Android will probable improve. The OS is maturing and hardware getting better, however the ecosystem of manufacturers and carriers will likely stay the same. The reality is that Apple is doing what Apple has always done and they've gotten really good at it.

Cooking with Chef slides

2011-07-25T00:00:00-07:00

A little late, but I have posted my slides from my talk at the East Bay Ruby Meetup in June.

Cooking with Chef

Check them out and feel free to ping me if there are any questions. At the meetup, was also talking about doing a bit of a blog series on getting started with Chef and posting some of the scripts and baseline setup I have used before. Hope to start forming some simple getting started resources.

New Opportunities

2011-07-19T00:00:00-07:00

Finally posting about it a little bit late, but last week was my first week in a new position with a new company. I truly enjoyed my stay at Involver, but saw an opportunity pop up and decided to take it.

This new opportunity is to be one of the first in-house Ruby developers at Demandbase. Demandbase specializes in a B2B analytics API for reporting and logging information about visitors hitting your business's website.

That all sounds fancy, but the gist is I'll be helping to grow and form the team, scale up and expand the platform, and able to leverage both skillsets as a developer and in operations. They've been working with the awesome guys from Highgroove, utilizing Chef in a continuous deployment setup, doing geographic load balancing, and have a pretty agile and responsive process setup.

I had an amazing time at Involver and grew so much personally and professionally. I was a part of some major projects and instrumented a lot of change. Our Operations team was top notch and put out of a lot of stuff considering it was only 3 people.

Going forward, have some goodies I'll hopefully be working on and open sourcing. There are a lot of interesting things in the works and hope to post about them over time.

And by the way, Demandbase is hiring. Hit me up @krobertson or shoot me an email.

Announcing Gemstache

2011-06-30T00:00:00-07:00

Pleased to announce Gemstache, a new service to enable teams and companies to build and distribute their own private gems with security and ease of use in mind.

Gems are an incredibly simple way to package up code and distribute it, making it available to multiple code bases or just a great way to encapsulate functionality. However, the way gems are traditionally distributed doesn't lend itself well to securing the gems and controlling access. The default "gem install" command you run doesn't provide any support for authentication against the gem source, so most sources are open to the public.

Gemstache works by giving you your own private gem source you can use. In order to download any gems from it, you must have the accompanied "gemstache" gem installed locally or on your servers. When talking to your private gem source, it interlaces to add authentication to the request as well as ensuring it is happening over HTTPS. With this, your gems are only downloaded over SSL and any interaction requires authentication.

To make managing your gems easier, the gem adds the ability to easily upload a gem to your private gem source and make it available. From your command line, just run "gem stache [your gem file]" and it is uploaded and ready for use.

Within the service, you can give users varying roles to define whether they can just download gems, are able to publish new gems, and are able to yank/delete gems.

Gemstache was born out of a need we had been facing at Involver, where some of our own internal code is enclosed in gems and used between a few of our codebases, as well as we have some gems we have forked to either add customizations or our own fixes. Traditional methods left them exposed and also more painstaking to release.

We're just about ready to launch a public beta. Visit gemstache.com and sign up to be notified.

Beware of error handling with fibers in an evented context

2011-05-26T00:00:00-07:00

One of the more interesting developments in Ruby lately has been the interesting things being done with Ruby 1.9.2, Fibers, and EventMachine. Whether you look at it being applied to existing libraries with rack/fiber_pool, or Goliath, it packs a punch and proving interesting for high performance evented applications.

Recently when reworking one of our backend applications to use rack/fiber_pool with EventMachine, I ran into a gotcha when it came to exception handling. Asynchronous applications makes heavy use of callbacks, where the context of when an operation is completed is not inline with when it began. Fibers work to make it easier to write asynchronous code look like synchronous, where you make a call, and it returns a result.

The problem is that this can play a trick on your eye. You think you are executing something in line, but the context it is operating within jumps without you being necessarily aware of it. Take the following example:

require 'eventmachine'
require 'em-synchrony'
require 'fiber'

def doit
  f = Fiber.current
  EventMachine.add_timer(1) { raise 'uhh ohh' }
  Fiber.yield
end

EM.synchrony do
  begin
    puts "beginning"
    doit
    puts "end"
  rescue Exception => e
    puts "ERROR: #{e.message}"
  ensure
    puts "Have a nice day"
  end
  EventMachine.stop
end

From looking at the main block, you have the call to doit wrapped in a begin/rescue, so it seems logic that if an error was raised within it, the rescue block would capture it and handle the error. Unfortunately, the error it being raised outside the context of the fiber, so it isn't caught. You are outside the context of the fiber in the time between when Fiber.yield is called and f.resume is called.

Why is this context important? Because when we built applications, we want them to be resilient. For instance, if a web request encounters an error, we want it caught, reported either to a log or a service, a response returned to the user, and then it be ready to handle the next request. However outside the context of the fiber, the error handling we've built in won't get called and the application can exit. Even if the application is monitored with god, existing connections will get closed out and connections won't be served while it is restarting, and the valuable information within the error is lost.

Once the fiber is resumed, the error handling functions as expected.

require 'eventmachine'
require 'em-synchrony'
require 'fiber'

def doit
  f = Fiber.current
  EventMachine.add_timer(1) { f.resume }
  Fiber.yield
end

EM.synchrony do
  begin
    puts "beginning"
    doit
    raise 'uhh ohh'
  rescue Exception => e
    puts "ERROR: #{e.message}"
  ensure
    puts "Have a nice day"
  end
  EventMachine.stop
end

In this case, the exception will be caught and handled, and you will make it to the "Have a nice day" message.

In order to handle the error, you can add some exception handling within the main context, or the root fiber.

begin
  EM.synchrony do
    begin
      puts "beginning"
      doit
      puts "end"
    rescue Exception => e
      puts "ERROR: #{e.message}"
    ensure
      puts "Have a nice day"
    end
    EventMachine.stop
  end
rescue Exception
  puts "We caught something outside a fiber!"
end

However, I haven't yet figured out how to have the same thing done in the context of a web app. So far, I've just ensured I have proper error handling in the callbacks before the fiber is resumed. Trying to put a begin/rescue block in the config.ru doesn't seem to have any effect on rescuing an error.

The moral of the story is when using fibers, be conscious of error handling outside the context of the fiber. Unhandled exceptions there can cause the entire application to exit. Be aware you have proper error handling when doing something that yields a fiber, and whether they fiber-enabled libraries you are using have error handling outside the context as well.

MySQL Read/Write Splitting in JRuby

2011-05-16T00:00:00-07:00

I was rather surprised about the lack of information out there about how to do read/write splitting with MySQL and Rails on JRuby. This is a pretty key part of our infrastructure, and has been a major point of our attention when performing large platform upgrades.

Having recently gone through one such upgrade and subsequent breaking of our read/write splitting, I had to dive into the code that was doing it more and get a better understanding of how it works.

With JRuby, there are two things to be aware of when trying to enable read/write splitting:

Does it route the queries to the right box?
Does it do it without sucking all available connections up on MySQL?

The second one sounds weird, however we have actually seen it and had it cause major issues. I am not totally sure what to attribute it to, but somewhere between ActiveRecord, ActiveRecord-JDBC, and the MySQL JDBC driver, there is some weird connection handling going on where a connection will still be established, however viewed as dead and a new connection will be opened up. Eventually, it just sucks up all available connections to MySQL.

This is a pretty big problem, but can work around it. Most of the existing information out there about how to do read/write splitting with ActiveRecord doesn't always apply cleanly to JRuby. In particular with one, we ran into the connection-sucking issue while in the benchmarking process.

Fortunately, there isn't a whole lot that needs to be done to enable splitting. The MySQL JDBC driver already has support for it with its ReplicationDriver, you just need to enable it when using Rails. Then you can leverage a plugin which sets a "read_only" property on the raw connection when it recognizes a 'SELECT' query. When read_only is flagged, the driver knows it can send the query to servers other than the master.

Steps to configure read/write splitting:

1) Get activerecord-jdbcmysql-adapter

If you're going to use MySQL on JRuby with ActiveRecord, this is rather a given. We're currently using v1.1.1 and happy. All the JDBC adapters are basically rolled up in one repo which produces multiple gems.

2) Get my version of the active-record-jdbc-mysql-master-slave plugin

There is a plugin that takes care of setting the read_only property on the connection for 'select' queries. The original version of the plugin was originally written for Rails 2.2 and hasn't been updated in 2 years. We originally used it on Rails 2.2.2 and had a lot of success with it.

It doesn't work on Rails 2.3 through, primarily because of the load order changing. It previously worked by alias_methoding the initializer on JdbcAdapter, and in its initialize, would alias_method the _execute method if using mysql. However, load order changed in Rails 2.3 and the database connection is initialized before the plugin is loaded, causing its own version of initialize to not run. I've updated it to use a slightly different method to ensure it happens regardless of load order.

To install the plugin, can just use ./script/plugin:

$ ./script/plugin install git://github.com/krobertson/active-record-jdbc-mysql-master-slave.git

3) Ensure active connections are reset in ActiveRecord

You can still experience some issues with stray connections. The best way we have found to combat this is through an after filter on controllers that does some cleanup. Add this portion to your ApplicationController:

after_filter  :clear_database_connections

def clear_database_connections
  ActiveRecord::Base.clear_active_connections!
end

By calling ActiveRecord::Base.clear_active_connections!, it ensures connections are reset. It is best to try and ensure it is the last after_filter defined.

4) Configure your database.yml

The key in the database.yml is to set up activerecord-jdbc to use MySQL's replication driver and to configure the "url" for it.

production:
  adapter:  jdbcmysql
  username: myuser
  password: secret
<% if $0 =~ /(rake|irb)/ -%>
  host:     masterdb
  port:     3306
  database: myapp
<% else -%>
  driver:   com.mysql.jdbc.ReplicationDriver
  url:      jdbc:mysql://masterdb:3306,slavedb:3306/myapp?roundRobinLoadBalance=false&autoReconnect=true&failOverReadOnly=true
<% end -%>

You may be looking at it and thinking I've got erb in my yaml... and yes. Rails lets you mix the two, and the reason in this case is because we opt to disable the read/write splitting in the Rails console (irb) and when running rake. Two very important reasons to. First, in console, we've had it suck up all connections just when doing simple debugging. Not fun. The normal adapter doesn't do it. And second, we want it off in rake for when we deploy and it does db:migrate. We don't want to allow replication delay in there, such as when querying the schema_migrations table.

For the read/write splitting part, the bulk of the settings are in the "url". The first host specified is the master, and any others are slaves. It is supposed to support multiple slaves and have the option to load balance between them, though we found in practice it never really worked. Instead we direct our web app and background processing to separate slaves. After the hosts comes the database name, and the query string parameters are a number of the driver options.

How the stuff works

At first, looking at the code for the plugin seemed rather confusing. Less than 100 lines of injecting into ActiveRecord did our read/write splitting? But the truth is it is fairly simple, as most of the work is handled by the underlying driver (MySQL's JDBC driver). All we really need to do is populate that read_property on the driver's connection. Shortened up, it does this:

alias_method :_execute_without_master_slave, :_execute
alias_method :_execute, :_execute_with_master_slave

# if we're in auto-commit mode and about to execute a select statement, 
# then set the connection in read-only mode for the duration of
# the query... which will permit the query to be load-balanced
# amongst the slaves by the mysql connector/j ReplicationDriver
def _execute_with_master_slave(sql, name=nil)
  # Need to set the read_only option on the raw connection
  # to tell the underlying driver whether the request can
  # go to slaves.
  cro = raw_connection.connection.read_only
  begin
    raw_connection.connection.read_only = 
      raw_connection.connection.auto_commit &&
      JdbcConnection::select?(sql)

    self._execute_without_master_slave(sql, name)
     
  ensure
    raw_connection.connection.read_only = cro
  end
end

To sum it up, the adapter has an _execute method which does the real execution and is passed in the raw SQL. The plugin injects itself in the middle. First, it captures the current read_only value, because after it executes the given query, it wants to ensure it is set back to that. Next, it sets the read_only property based on whether MySQL itself has auto_commit on and if it is a 'select' query. It then calls the original _execute method, and after it runs, sets read_only back.

The actual plugin has several additional things, like allowing you to run some statements within a block to ensure it goes to the master, and to only work with the MySQL adapter.

Transparency and status

2011-04-29T00:00:00-07:00

One thing that I think can really be highlighted from the recent AWS issues is the value of a status dashboard for your platform/service.. However, simply having one isn't quite enough. Plenty of services already do. What really matters is how effectively it is used.

Just think why a customer would be coming to your status dashboard and what they'd be looking for. If they're coming to it, it is likely because they're having a problem. What they'd be looking for is confirmation that the company is aware of the issue and a rough idea of when it would be fixed.

This sounds quite simple and plainly put, however rarely done. I don't know if its some management/PR intervention of not wanting to sound unstable or unreliable, or if it is just that level of transparency isn't encouraged/practiced at these companies, but simple information often never makes it public, or seems to after the fact.

One our major gripes with Amazon when we were hosted with them was how we'd become aware of an issue, but in some cases it would be hours before an acknowledgment of it went onto the status dashboard. The 1am EBS outage of last week has repeated itself before, and interestingly around 1am several times. It affected Reddit just a month or so ago, and it affected us back in October. One of our other guys was up at 1am, aware immediately of the EBS issue since it was impacting our master database server. It took Amazon two and a half hours to post about it on their status site, even stating it began the same time we were paged.

An additional aspect often missed is the value of the post-mortem. When you have a major issue, customers lose confidence. The post-mortem is a chance to try and gain some of that back. It doesn't need to divulge every single detail and shouldn't look at it as an embarrassment. When I think of a post-mortem, I look for 3 things: what happened, why it happened, and how it won't happen again. It can very effective at reassuring users if you can clearly show that you understand what went wrong and have a solid plan to ensure it won't happen again. Glossing over the chance by simply saying "we've fixed it" (without saying what it is, in some fashion) can simply detract users even more.

Example of a post-mortem done wrong? Look at Tumblr's downtime in December or their security leak in March. Look at the majority of AWS issues. Typically, they don't do them. They only do them when they majorly screw up. And when they do, there is an air of "architect for us" attitude.

Here are a few quick things that would make any status dashboard serve its purpose well.

Update early, update often.
Don't downplay the severity.
Give ETAs on resolutions.
Be very generous with ETAs (go worst case).
Choose a good layout! Look at the image to the side for Amazon's site during their issue.

Above all though, look at the status site from the perspective of their users. Think about why they would come there and what information they're looking for.

Talk Cloudy To Me

2011-04-27T00:00:00-07:00

I will be talking at the Silicon Valley Cloud Computing Meetup this Saturday, April 30th. If you are interested in the topic and in the area, please come out! So far there are about 500+ people plus registered to view panels discussing a wide variety of cloud-related topics throughout the day.

I will be speaking on the panel titled "What are public clouds still missing?" Will be talking about what areas the public clouds still fall short in and where the road ahead will hopefully take us.

Other speakers who will be there include notable people from VMware, including some of the CloudFoundry engineers, Netflix, Cisco, Microsoft, and Rackspace.

Definitely come and check it out! Saturday starting at 1pm, hosted at the Microsoft Mountain View offices.

Amazon and what it means to you

2011-04-22T00:00:00-07:00

Unless you were under a rock, you probably heard about the major issues in Amazon's US-East region yesterday. Needless to say, a lot of businesses had a rough day yesterday.

One of the most interesting aspects of yesterday was realizing how big Amazon's reach is. Numerous well known startups and businesses were down, but beyond that, services not even on Amazon themselves were affected. We moved off EC2 back in December, however we integrate with APIs from several other services, including some that are running on EC2. We still felt the impact and were closely watching Amazon tracking status.

The event lead to some interesting discussions though.

There has been a lot of blame going back and forth, with some leaning on the side of "you should have planned for it". While it can be particularly harsh, in that most best practices for Amazon stressed leveraging multiple availability zones, while yesterday, it was nearly an entire region that was out. There is an element of truth in the argument... have a plan.

Typically that is known as a Disaster Recovery plan. They easy to be overlooked until you realize you need one. They're always one of those "we should do that someday" or maybe even something a company doesn't think its big enough to need.

I am no expert on disaster recovery (and there are plenty of consultants out there to help build a professional one), but there are some pretty basic things you can look at to at least have a plan rather than no plan. Some services recovered pretty well yesterday, albeit with partial functionality, but they got something back up. Others realized they were at the mercy of another provider and were stuck until then.

There are some simple questions you can begin to answer for yourself for a simple plan:

What data is important? Your service runs on code, but it is powered by data. What data is it dependent on? Image assets, file uploads, databases?

Where does my data live, and how can I have it in multiple places? Take regular database backups and store them off site. Keep a backup of assets. It is important for them to be in more than one location since if that location disappears, you are SOL.

How much data loss is acceptable? If stuff goes down, you may lose some data. If you want to limit the loss, you need to keep fresher copies of data. More frequent backups, perhaps a live DB slave hosted somewhere else. You need to either accept that some data loss is acceptable, or architect so that you won't have any data loss.

How fast can you provision servers and where? Geo distribution all the time can be expense and not necessary (unless you deem it is), but being able to have somewhere to quickly set up emergency servers is a major plus. If you use AWS East, then maybe use AWS West for DR. Make sure your account limits give you enough capacity, and have scripts and basic stuff in place and ready to be used. We leverage Chef for all our systems, so we can quickly launch some new servers and get right to work configuring them pretty quickly.

What is your order of priorities? A service is often comprised of many smaller components. Know which ones are the highest priority. Know which functionality without your app is most important. You'll be racing against the clock and have only so many people. This lets them focus their attention to help get core functionality up as quickly as possible.

Do a fire drill. This is something I think we'll be doing pretty soon. Out of the blue one day, do a mock fire drill. You are down, your datacenter disappeared. Get the service back up in an alternative place. How long does it take you? What were the pain points? Beyond simply having a plan, you need to know the plan will work.

And lastly, know your application. Know its requirements, constraints, and understand its bare minimums, most important pieces, etc.

Power through environments

2011-04-14T00:00:00-07:00

At Involver, we have a pretty rich deployment environment that I think isn't very common. It is normal to hear about places having a separate staging environment from the production environment, and sometimes having a staging and test environment. We actually go a step further and have a total of 8 non-production environments. They all serve different purposes, but are all heavily used.

Our environments are broken up like this:

Production: duh, live stuff
Staging: Used for QA, close to production, no experimental stuff, pretty good data set and history in the DB.
Demo: Used for client-facing stuff that is under development. Somewhat experimental, but expected to not be breaking or of a major impact.
Perf: This environment is regularly repurposed and is used for benchmarking, testing infrastructure changes, and other large scale changes (Rails upgrades, DB upgrades, core app perf testing).
Test1-5: These guys are pretty special.

The test environments are something special and are meant to be highly flexible. They are used to deploy separate git branches to an environment that has all the different elements of our application fully orchestrated. They are useful for testing volatile code, experimental code, or for performing QA on features or feature branches before the code gets merged into master.

The deployment to these environments works by specifying the branch on deploy, and it will create the MySQL/Mongo databases, setup the DB schema, bootstrap the DB with some initial data, and then configure all the configuration files and even config files for other applications that it interacts with. Each branch on each environment gets its own database. And between deploys, the data for your branch is fully preserved.

So say you deploy "dev-fix-booboo" to test3 for the first time. It sets you up with a fresh dataset and are good to go. You do your thing, then later on, someone deploys another branch. They get their own DB, not mucking up the stuff you previously had done, and when you deploy there again, you are right back where you left off.

Since they are specific to the environment too, we often times assign a long-running feature branch to an environment, so that the data used by QA sticks around more, and smaller test branches can use them here and there.

The full power of these comes form:

Flexibility. I have a branch I want to deploy, but not much other environments. Sweet.
Some element of persistence. I can use it for a few days, come back, and be right where I left off. No risk of someone polluting my stuff with bad data.
QA can actually deploy to them and pick which is available. For non-feature branch testing, this gives them great flexibility.

Of course, there are still some pitfalls:

They don't mirror the full production setup. They're usually one box and not spec'd for any kind of load. They run good, but not going to handle lots of people. Can't perf test here.
There is the data persistence between switching branches, but often no rich DB history. It is harder to test large datasets of legacy type setups.
Can't test infrastructure changes. If deploys don't switch out gems or system configs, so harder if you want to test a change like that, you need to lock the environment til done.

All of this makes for a very flexible deployment process. It allows devs to easily get code off of their machines and in a more controlled environment. It allows us to have high confidence in code before it makes it to master and often even before the feature branch. It also allows QA to do their testing without potentially deploying code they reject to staging or demo, where it might be customer facing or impact other development.

Big props for it go to @mikewadhera, who initially wrote the deployment process for it. Since then, we (infra/ops team) have worked on making the environments single-system, which allow us to have more of them, as well as iterate on making them more streamlined with developer and QA feedback.

This kind of builds up to something I'll be talking about soon too. Since we have a total of 9 different environments, this can lead to a very interesting chef workflow.

Serving Static Content Via POST From Nginx

2011-04-12T00:00:00-07:00

Recently, a change was made on Facebook's end where their <fb:iframe> FBML tag started doing POST request to get content. Their documentation seemed to indicate their "Post for Canvas" change wouldn't affect the <fb:iframe> tag, however our error logs spoke quite to the contrary. We saw an influx in errors in places where we used <fb:iframe> to pull in static content. Nginx would simply return a 405 error for "Method Not Allowed". Nginx can't serve static content on a POST request.

Some quick googling seems to reveal some quick answers, but the problem is none of them worked.

The most popular, this thread, talks about a work around by manipulating the error page for nginx. First, there is some mixed syntax depending on the nginx version. Some references seem to be to old 2007 syntax which doesn't work on the nginx 0.7 build we use. But they all indicated that this should have worked:

error_page 405 =200 @405;
location @405 {
  root ...;
}

This basically tells nginx to change the response code to 200 for 405 messages, and to use the @405 location entry to handle it. Setting the root it is supposed to pull the requested document. It didn't work though. At best, I got it to return a 200 code, but no response body. Essentially, it was still doing a POST request and it still didn't like it.

I could however, get it to be served by proxying it to our Rails application, if I enabled serving static content from Rails. But that was less than ideal.

But then, I was scanning through nginx's proxy documentation and found what seemed like an ideal solution. I could configure nginx to serve static content on another port, have the main server proxy it to the other port, and when proxying, change it into a GET request:

upstream static_backend {
  server localhost:89;
}

server {
  ...
  error_page 405 =200 @405;
  location @405 {
    root ...;
    proxy_method GET;
    proxy_pass http://static_backend;
  }
}

# POST STATIC CONTENT
server {
  listen 89;
  server_name _;
  root ...;
}

Our pain points with EC2 and how our moved solved them

2011-02-16T00:00:00-08:00

As I mentioned in my last post, we made the move off of EC2 in December to our own cluster of machines.

While EC2 had served us very well and helped get our systems to the place they are today, we started reaching a point where we were running into three main bottlenecks with the EC2 ecosystem and we couldn't find any way around them.

Will start off by saying the AWS is an awesome service and we couldn't have grown to the level we are at today without them. Amazon should be a easy contender for any service just starting out. Some of the services they provide, especially things like Elastic Load Balancer (ELB) and Elastic Block Storage (EBS) are services that are simple to start using, help tremendously as you grow, and simply aren't common with traditional VPS hosting.

However, here are some of the bottlenecks we faced and how we've solved them in our new setup.

IO Performance

Anyone who is doing anything intensive on EC2 will tell you that IO performance is horrendous. We were battling EBS constantly on our database servers and had serious questions how we could continue to operate yet alone scale our DBs while in the cloud. We met with AWS engineers to review our configurations, but were already doing all the best practices. We had tuned MySQL configuration as much as we could, had the "high-memory" instances and higher IO priority and newer CPUs, and were running 4 EBS volumes in RAID10.

They tried to push Amazon RDS, however RDS is simply EC2 and EBS with the same best practices. While it might be slightly better, it wasn't going to totally solve our problems.

The biggest issue was IO latency. We had database servers where it was normal for them to average 20-30% IO wait. We frequently had spikes upwards of 60% to sometimes 90%.

And it wasn't always disk performance, but IO as a whole. We would at times provision a new database server that would simply struggle to stay fully replicated with no query load on it. Either struggling with IO latency, or sometimes even keeping a connection up to the master.

There is unfortunately no solution. Everything in AWS is a shared resource. With that comes variable performance and bad spikes due to hot-spots. It is difficult to predict or ensure consistent expectations.

Support

At one point, we encountered an issue where we were seeing extreme IO latency on our EBS volumes on our master MySQL DB server. One of our other ops guys was up for several hours in the middle of the night, along with our CTO and Director of Product. Unfortunately it was an EBS issue throughout the entire zone and we had a large footprint in that zone, including our master database server.

The biggest thing we took issue with was that it was almost 3 hours before it actually made it to Amazon's status dashboard.

After that incident, we started looking at Amazon's Premium support, however the support packages are really quite weak when you dig into them.

First of all, Premium is a straight 20% added to your bill. You can't select it per instance or anything. We didn't care about premium support on our staging systems, so we'd have to move those to another account if we signed up.

But the big thing... you get 24/7 phone support and a 1 hour acknowledgement SLA. Meaning they'll confirm you are having an issue within 1 hour. However, resolution is another thing. Because everything in AWS are shared resources, they can't offer any sort of priority resolution at all.

So that EBS latency issue? It took 3 hours to make their status dashboard, and a total of 11 hour from when we noticed it to them posting it was totally resolved.

Paying for Premium support wouldn't have helped much. It would still be fixed for us at the same time. At telling the CTO/Director "AWS has acknowledged it, it'll be fixed when its fixed" isn't the type of message we like to deliver.

Scaling Costs

Amazon is great for services just starting out. It is simple pay-as-you-go, you get availability of a lot of nice functionality with S3, Elastic Load Balancer, and others, and it is quick to get bigger instances as you grow and need more.

However there comes a threshold where the costs start growing more quickly than your scale.

Want support? Bam, extra 10%-20%.

IO bottlenecks? Amazon's answer is to use bigger instances (higher IO priority) and spread the load across multiple instances. Both of those mean more money.

Once you start reaching bottlenecks, the only answer within AWS is to work around them, and that usually means more costs.

Additionally, since it is so easy to provision new systems, it is easy to get carried away without realizing how much the changes are costing you.

Insight

This is actually a bottleneck we didn't really realize until after we moved. It is amazing how little insight you have into your architecture until you have full control over it.

For instance, on the new setup we had nice zone separation between systems and were seeing massive spikes in concurrent connections through the firewall, 3x more than we expected. They all ended up being DNS traffic. Since it was UDP, the "connection" count was skewed, but we hadn't realized how much DNS traffic we generated. In AWS, we couldn't monitor low level traffic metrics in the same way, track our aggregate DNS traffic, or anything.

Our Solutions

Our solutions to all of these were actually quite simple and are a nice blend between old school architecture and modern pragmatism.

Nowadays, you start to hear all this stuff about "private clouds". First of all, drop the bullshit. "Cloud" is slapped on anything it can be these days. There is nothing new to the concept of "private clouds".

All it is is a virtualization cluster on equipment dedicated to you. Its been around for 10+ years. Clouds caught on because they were simple to get started with and great at the small scale. Now its just a marketing buzzword.

Our approach was simple. We separated our environment out into a hybrid of dedicated systems for some things, and a large SAN backed virtualization cluster for everything else.

Databases

Dedicated hardware is the perfect solution for IO intensive workloads. With this, we could actually specify what we wanted. Types of disks, RAID configuration, capacity, etc. We could go as simple as RAID 10, up to SSD or even FusionIO.

We moved all our MySQL and MongoDB systems to dedicated hardware, each built to their own needs and planned ahead for a certain timeline. We then looked at several growth metrics. For instance, MySQL would likely need more CPU or disk capacity in the future. MongoDB would likely need more RAM in the future.

Virtualization

Almost everything else was virtualization. We got several very big, bulky virtualization systems. 12 cores, 24 threads, 128gb RAM, ohh yeah. However we slice and dice it is up to us. We have everything configured for full failover of the base chassis.

At the SAN level, we left that up to the guys at Network Redux. They monitor the workload levels and plan for it accordingly. The IO doesn't scale infinitely, but rather we try to maintain an average expectation so that growth is predictable.

Support

Support is probably one the biggest and most dramatic changes. AWS is a complete black box. Now, we have two people dedicated to our account, as well as a 24 support desk we can call and have a tech pull up all the details on our account. We have the cellphone numbers of the people dedicated to our account. We have yet to call them at 2am, however having it is more than we ever had with Amazon.

We now have people proactively working to help us. If IO load is growing on the SAN, they reach out to us ahead of time, before it becomes a problem on our end.

If a hardware node throws an alert, they get paged at the same time and will hop online to help us to investigate.

This is probably one of the biggest pros and our new provider, Network Redux has been truly awesome in that regard. They are proactive, we deal with the same people over time so they are familiar with our deployment, our change history, etc.

Costs

Costs though... surely all this raw hardware goodness, scalable IO, and choice in your deployment costs a whole lot extra, doesn't it? Not so. You might actually be surprised how much cheaper it is. I can't give numbers, but I will say it was significant and after the move one of the main bullet points added by the CFO was how much cheaper it was.

To make the CFO even happier, we had planned ahead for growth, where we could handle spikes and add some additional systems without extra costs, and had solid figured on how much the costs would increase. "When we need more capacity here, the next step is X, and the cost will be $Y." As more time goes on, we're getting better at predicting when the growth will be needed, and showing the benefits.

Conclusions

Overall, Amazon is great service. Don't get me wrong. It is truly excellent for companies/services just starting out and has an awesome growth path for scaling early on.

However nothing scales infinitely. Not your applications, and not the benefits of a service like AWS.

Once you each a certain threshold, you benefit more from full control over your environment and you get to the point where it is actually cheaper than the cookie cutter services AWS provides. Where that threshold is depends on each case. But it is important to know where it is, even if you aren't there yet.

For us, we improved our performance several times over, have more control than ever, and did it all while actually saving money.

How we did a datacenter migration with no downtime

2011-02-11T00:00:00-08:00

Starting in October, we at Involver decided to take on a project migrate our entire infrasture off of Amazon EC2. AWS had served us well, however there comes a point where the limitations of some of their services were becoming a major bottleneck. The bottlenecks and how we've resolved them in our migration is a big enough topic for a follow up post, but we recognized that we needed more control over our environment and the specifications of all the systems.

Update: Have posted the follow up on the bottlenecks we were facing EC2.

After nearly a month of evaluating providers, doing benchmarks on test systems, and milling over all the details, we picked our provider. We then spent another month refactoring various systems, re-evaluating all machine roles, and system configurations. Since we were no longer constrained by EC2 instance types, we started looking at how many CPUs and how much RAM is really best suited for each system.

For our hosting provider, we ended up going with Network Redux. Of all the others, both big names you'd recognized and some smaller ones you wouldn't, they proved the best fit. Go to any provider and they can all get you the same server. That is easy. Its the other services, the skills of their staff, and quality of the total, big picture architecture that matters. With Network Redux, a lot of it came down to the fact they they worked like us. They hired top-notch people, they moved quickly on things, and had a short decision tree. Case in point, when the President is on your sales call and can say you'll have a test system with set specs in two days, vs the sales person needing to pull out an org chart and "get back to you."

After our month of refactoring, we finally flipped the switch the night of December 23rd. We did an entire datacenter migration with no user downtime. Even more, we didn't even have all of our production systems until that morning (thanks Dell! </sarcasm>).

So how did we pull it off?

It boils down to an excellent team, delicate planning, and an awesome toolset. Our operations team is only 3 people, but we manage what is now 8 environments where our code lives and about 100 systems.

Everything in our environment is fully configured with chef. With the migration, we refactored a lot of our recipes put a strong emphasis on everything in chef. Each system was provisioned, cooked and recipes tweaked, then destroyed, reprovisioned, and recooked. Repeated until it was ready to go right after cooking.

MySQL

Our production database servers were the first to get set up on the production side. We'd already validated all our chef recipes for both our MySQL and our MongoDB clusters on our staging environment beforehand. With MySQL, we run a master-master setup with multiple read-only slaves. We cooked each of the boxes and configured all of the slaves to point to what would be the new master. We made a change to ensure that the new master and the EC2 master had different auto-increment offset and we also ensured log_slave_updates was enabled. We then imported a backup of our production database onto the new master. It automatically replicated out to the passive master and read-only slaves. We then set it up so our new master would replicate from our EC2 master.

With this configuration, writes would be coming in on our EC2 master, get replicated to the new master, and since it has log_slave_updates on, those changes would then replicate our to the other database servers. This also enabled us to switch over the site and easily allow writes to start coming in to the new environment. If there were some lagging requests to the old site, they would replicate to the new environment without any collisions. We also documented the position of the new master when the switch happened, so if we needed to revert, we could potentially set the EC2 master to replicate from the other master.

MongoDB

The MongoDB databases were migrated in a different fashion. We were moving from a single replica set configuration to a sharded setup with two replica sets. Because of this, we could do the same early replication process we did with MySQL. Mongo is primarily used for our analytics data, so with it, there is a primary collection that is most important and then others used for the calculations. We cooked the database servers ahead of time and got the sharding configuration and everything fully setup before hand. Then the data migration was scripted out so that with a quick command, we would pull the most important data over first by exporting the data, transferring it, then importing it. This way, it was migrated and sharded at the same time and fully automated.

Our Application

The morning of the 23rd, we came in, having just had the bulk of our production virtualization cluster setup the night before (thanks again for the delay Dell). This is where the awesomeness of chef and our team truly shined. We set to work and provisioned all of the production systems, cooked them, set them up in load balancing, verified security rules, and completed some of the final code tweaks and testing. We didn't even deploy our final monitoring systems until that morning. We did a few test deploys, verified all the parts of the app would come up, and that monitoring on them was accurate.

The Switch

Finally, around 10:45pm, we made the switch. The actual cut off boiled down to just a DNS change. TTLs had been changed to 60 seconds weeks earlier. We just repointed the necessary hosts and within 60 seconds, we saw the majority of traffic cease on EC2 and come in on our new systems.

Right after we made the DNS change, we also removed all of our old app servers from ELB (Elastic Load Balancer) except for two. We turned nginx off on those two and turned on rinetd, which was already configured to forward any traffic to port 80 or 443 to our new environment.

The benefits of the migration were almost instantly visible. Noah, our CTO, was in the office with us to do acceptance testing right afterward. We made the switch, verified DNS had updated, and when he loaded the first page, his first reaction was "Holy shit... this is so much faster!" That there was all the validation we needed for the 2 months of work we had put in.

Most impressive was looking back at data in New Relic from before and after the migration. It was quite surprising how drastic the improvement was.

And its gotten even better. Having to no longer fight fires with AWS anymore, we've actually made tons of other enhancements.

How a deprecated method can take down a process

2011-02-07T00:00:00-08:00

At Involver, we have a pretty active background processing tier that handles a variety of things from long running tasks we don't want in the web app, pulling in content from external sources, and simple recurring maintenance tasks.

Previously, we ran a single worker thread per JVM, but that setup lead to a low worker per system density since it sucked up a lot of memory. A few months ago, we came up with a way to run multiple workers within a single JVM and have it nicely thread safe by using a small embeddable JRuby VM started off the main process. This worked excellent and doubled the number of workers we could run per machine, since they could share heap space.

But then we recently began experiencing an issue where all the threads would just halt processing. No errors, seemingly nothing wrong. Heap usage was fine and a thread dump showed them seemingly ok, but they just sat. We tried attaching a profiled, however it ended up just output garbage when we tried to have to dump its data after it got stuck.

After 2 days of investigation, finally managed to tracked the cause down to the fact that a change a few days before had caused us to start running over a function that used a deprecated method call, and the change meant we were hitting that line 500-1000 times per minute.

At first, we suspected maybe JRuby had some sort of thread concurrency issue with calling deprecated methods, but that made no sense. All calling a deprecated method does is write to the console!

As Gru in Despicable Me would say... "Light bulb!"

When running a single worker in console, we saw the warnings, but when running the multiple workers per JVM, we saw none. The rabbit hole gets deeper and fortunately lead us to the solution and were able to confirm the issue and resolution.

The way we spawned the child threads, we found stdout/stderror was not attached to anything. They make use of buffered IO though. There is a fixed buffer size, and if you try writing more than that, it will block on writing to it until the buffer is read from and space is freed. Since nothing was reading from it though, the threads would simply block indefinitely on writing out the deprecated method calls.

So we set up a test. We created the following class on one of our boxes and spawned up the multi-worker process.

class Awesome
  def self.kill_me_softly
    loop do
      puts "a" * 1024
      Rails.logger.info 'Alive!'
    end
  end
end

It came up, but would get hung in less than a second. Simply do no more. One threads would lock up independently, one by one. So the whole JVM wasn't frozen, just that child thread. Hence why they all died at different times, but very close together.

Then to confirm even more and fix it, we changed the multi-worker process to reopen the stdio handlers on child threads:

STDIN.reopen "/dev/null"
STDOUT.reopen "/dev/null"
STDERR.reopen "/dev/null"

Spawn one back up, and it begins streaming to no end, no lock ups, no problems. Success!

In the end, we changed the function to no longer use the deprecated method, and had the multi-worker process redirect stdio for child processes to an actual file, so that way we can actually log any valuables messages we might be otherwise missing. But that is hopefully the first and only time something as off as stdio buffering takes down a process.