In Valid Logic

To follow up on my previous post about disk errors with Ubuntu 12.04 on XenServer, I wanted to cover the process I've put in place for provisioning Ubuntu VMs.

With PaaS.io, I have a mixture between systems deployed on bare metal and virtualized. With the virtualized systems, I set out to make the provisioning as easy as if I was using an IaaS provider, while still giving me fine control over sizing and placement. Some of the decisions there are enough for another post.

Ubuntu 12.04 is the new hotness and I'd been anxiously awaiting it, with plans to progressively roll it out throughout PaaS.io.

With XenServer, it provides an easy template for provisioning Ubuntu Lucid 10.04 VMs and even for some newer releases. However, the way XenSever provisions Ubuntu is to essentially netboot it and install it from a remote source. Because of that, we can't simply use one of those templates buts install a newer release.

Fortunately, the template itself is very simple, and which release it installs is just a configuration parameter.

To create a new Ubuntu 12.04 template, simply log into the XenServer console and runt he following commands. It will clone the Lucid template and then change the parameter for the release to install from lucid to precise.

$ TEMPLATE_UUID=`xe template-list \
        name-label="Ubuntu Lucid Lynx 10.04 (64-bit)" params=uuid --minimal`
$ NEW_TEMPLATE_UUID=`xe vm-clone uuid=$TEMPLATE_UUID \
        new-name-label="Ubuntu Precise (64-bit)"`
$ xe template-param-set other-config:default_template=true \
        other-config:debian-release=precise uuid=$NEW_TEMPLATE_UUID

Now you will find a "Ubuntu Precise (64-bit)" option in the template list of XenCenter.

Now, we can actually provision a new box. Next, one interesting discovery was that you can pass in a kickstart script in the boot parameters options for the new VM.

Kickstart allows you to do a scripted installation, automated pretty much every aspect of the system. Instead of picking a bunch of options, it allows for an easy, repeatable process that basically leaves the machine completely ready to go.

To use a kickstart script, make a script available over HTTP somewhere on your LAN or on the public internet. By the time it is grabbed, the machine will have an IP, DNS, and all. Then, simply add ks=http://url/to/your/kickstart to the beginning part of "advanced OS boot parameters" option when selecting the installation media.

Below is a cleaned version of the kickstart script I used on my Ubuntu VMs. The main things of note:

• Sets up my partition table
• Configures base system with ubuntu-minimal as well as some common packages like openssh-server, curl, wget, and screen.
• Disables the creation of the ubuntu user (I create a normal every day user through chef)
• Configures the fstab with barrier=0 as mentioned before
• Disables /bin/sh from pointing to /bin/dash (personal preference)
• Updates apt sources
• Installs xenstore-utils
• Downloads some auto-configuration scripts
• Installs XenTools
• Installs the Ubuntu virtual kernel and removes the generic one
• Cleans up apt caches
• Shuts down the VM.

lang en_US
langsupport en_US
keyboard us
timezone America/Los_Angeles
text
install
skipx
halt
url --url http://us.archive.ubuntu.com/ubuntu

rootpw pa$$word   # you should replace, and use --iscrypted
auth --useshadow --enablemd5
user --disabled

bootloader --location=mbr
zerombr yes
clearpart --all --initlabel
part /boot --fstype=ext2 --size=64
part swap --size=1024
part / --fstype=ext4 --size=1 --grow

network --device=eth0 --bootproto=static --ip=10.0.0.50 --netmask=255.255.255.0 \
        --nameserver=10.0.0.1 --gateway=10.0.0.1
firewall --disabled

%packages
ubuntu-minimal
openssh-server
screen
curl
wget

%post

# update fstab for the root partition
perl -pi -e 's/(errors=remount-ro)/noatime,nodiratime,$1,barrier=0/' /etc/fstab

# point sh to bash instead of dash
rm /bin/sh
ln -s /bin/bash /bin/sh

# add normal apt source list
(
cat <<'EOP'
deb http://us.archive.ubuntu.com/ubuntu/ precise main restricted universe
deb http://us.archive.ubuntu.com/ubuntu/ precise-security main restricted universe
deb http://us.archive.ubuntu.com/ubuntu/ precise-updates main restricted universe
EOP
) > /etc/apt/sources.list
apt-get update
apt-get upgrade -y

# install some additional packages
apt-get install -y xenstore-utils

# set up xenserver automation scripts
AUTOMATER_REPO=https://raw.github.com/krobertson/xenserver-automater
curl $AUTOMATER_REPO/master/usr/sbin/xe-set-hostname > /usr/sbin/xe-set-hostname
curl $AUTOMATER_REPO/master/usr/sbin/xe-set-network > /usr/sbin/xe-set-network
curl $AUTOMATER_REPO/master/usr/sbin/generate-sshd-keys > /usr/sbin/generate-sshd-keys
curl $AUTOMATER_REPO/master/etc/init/xe-automator.conf > /etc/init/xe-automator.conf
chmod a+x /usr/sbin/xe-set-hostname
chmod a+x /usr/sbin/xe-set-network
chmod a+x /usr/sbin/generate-sshd-keys

# setup locales
locale-gen en_US.UTF-8
update-locale LANG="en_US.UTF-8"
echo 'LANG=en_US.UTF-8' >> /etc/environment
echo 'LC_ALL=en_US.UTF-8' >> /etc/environment

# install xe tools
cd /tmp
wget http://some/url/to/xe-guest-utilities_6.0.0-743_amd64.deb
dpkg -i xe-guest-utilities_6.0.0-743_amd64.deb

# install paravirt kernel image
apt-get install -f -y linux-virtual
dpkg -l | grep generic | grep linux | awk '{print $2}' | xargs apt-get remove -y

# clean up some stuff
rm -f /etc/ssh/ssh_host_*
rm -f /var/cache/apt/archives/*.deb
rm -f /var/cache/apt/*cache.bin
rm -f /var/lib/apt/lists/*_Packages

It is important that it shuts down at the end. My goal was to have it be all inclusive, and that means setting up the virtual kernel. However, it can't successfully reboot, because we need to update some PV boot options before it is ready to go.

Also, some packages must be installed in the post steps. The default install sources for Ubuntu don't always have all packages available, and I found it best to do the kernel last.

When you first bring up the new VM in XenServer, you may need to enter in a few details if you don't have DHCP running. It will self configure by default, but I normally opt to manually configure it to ensure the template gets a certain IP just to avoid any future possible collision.

After the bootstrapping is done and the VM is then off, log into console on the XenServer host itself and run the following snippet:

$ VMNAME=precise-20120501
$ UUID=`xe vm-list name-label="$VMNAME" params=uuid --minimal`
$ EDITOR=cat xe-edit-bootloader -n "$VMNAME" -p 1 \
        -f /grub/grub.cfg > /tmp/$VMNAME-grub
$ KERNEL=`grep vmlinuz /tmp/$VMNAME-grub | grep virtual |
        grep -v recovery | awk '{print $2}'`
$ ROOT=`grep vmlinuz /tmp/$VMNAME-grub | grep virtual |
        grep -v recovery | awk '{print $3}'`
$ RAMDISK=`grep initrd /tmp/$VMNAME-grub | head -1 | awk '{print $2}'`
$ xe vm-param-set uuid=$UUID PV-bootloader-args="--kernel=$KERNEL --ramdisk=$RAMDISK"
$ xe vm-param-set uuid=$UUID PV-args="$ROOT ro quiet console=hvc0"

Before you run it, of course set the VMNAME to the name of your VM. You might also want to check some of the values as you go (echo $UUID). You can use the xe-edit-bootloader command to view the grub configuration within the VM, but need to set the PV settings outside the guest. By faking the EDITOR to cat, you can export the grub file to a local file, then use some grep+awk to get the necessary pieces and finally set the PV settings correctly.

At this point, the machine is ready to be booted, converted into a template, or simply cloned. I personally like to leave my templates as never-been-used.

One of the main benefits of an IaaS setup like OpenStack or CloudStack is the self orientating of the VMs. Normally with a template, it keeps the template's setting for its hostname and network configuration. However, I found some example scripts on Github for passing network information into a VM through the VM's "xenstore" options. That way on boot, the VM can read in the settings it needs and update itself. I did some work to update the scripts to work with Precise, to more thoroughly update the hostname and DNS, as well as to regenerate the ssh host keys (since the kickstart did delete them).

As an example, this would be how to clone the template to a VM, set the params, and boot it:

$ UUID=`xe vm-install template=precise-20120501 new-name-label=builder`
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/ip=10.0.0.11
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/gw=10.0.0.1
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/nm=255.255.255.0
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/ns='10.0.0.1 10.0.0.2'
$ xe vm-start uuid=$UUID

That is cool and all, but I don't want to log into the XenServer console each time I want to setup a VM. Luckily, found a plugin for Chef's knife utility that adds XenServer provisioning. I forked it and added setting xenstore network parameters to it.

Now, provisioning a new VM, configuring it, and bootstrapping chef on it is just a matter of one call:

$ knife xenserver vm create --vm-template precise-20120501 \
                            -x root --keep-template-networks \
                            -r "role[foo]" -E production \
                            --vm-ip 10.0.0.12 --vm-name foobar

The command may be a little long, but it is all encapsulated in a single command. I simply have an internal README of sorts with the command pre-prepared for various roles.

If you have any questions, please leave a comment and ask. I'd be glad to help and always looking to improve my process as well.

Ubuntu 12.04 LTS Precise is hot off the press right now. Over the past few days, have been working on building new base images for PaaS.io, but was running into random issues where the root parition would encounter an error and freeze up. It normally happened just after it would finish booting and about 50% of the time.

A few times it happened after I already logged in, so was able to do basic read only operations. In dmesg, was able to see the following:

[    6.748868] blkfront: barrier: empty write xvda op failed
[    6.748876] blkfront: xvda: barrier or flush: disabled
[    6.748890] end_request: I/O error, dev xvda, sector 6584768
[    6.748908] end_request: I/O error, dev xvda, sector 6584768
[    6.748943] Aborting journal on device xvda6-8.
[    6.767022] EXT4-fs error (device xvda6): ext4_journal_start_sb:327: Detected aborted journal
[    6.767046] EXT4-fs (xvda6): Remounting filesystem read-only

Or if you weren't able to log in first, you might see this in the XenCenter console:

After some Googling, was able to track a similar error down by someone else. The fix was to update the mount options for the root partition. Mine are now:

noatime,nodiratime,errors=remount-ro,barrier=0

The key is the barrier=0. From some documentation, it is an option to help increase the integrity of writes by ensuring everything is flushed to disk be committing to the journal. However sometimes in a virtualized environment that is difficult to guarantee. In my case, have disk->RAID->dom0->LVM->domU.

Figure many other people will be diving into Precise this weekend, potentially running into this issue like me.

Soon, will post some additional details about how to easily get a nice Precise template in XenServer 6. I've been getting my setup nicely tuned using a kickstart script for the base system and leveraging xenstore data to dynamically setup the IP and hostname on boot.

While working on developing PaaS.io, one of my primary objectives is to ensure that everything is configured to be truly Highly Available (HA). This means everything has a secondary, replicated, and has automated failover. This way it is nicely resilent and fault tolerant. Core infrastructure changes can be made without any affect to running applications, servers go down without issues. Sweet, huh?

When it came to setting up MongoDB, the obvious option was to go to using Replica Sets. They're like master-slave replication in RDBMSes but the master node is essentially floating. Client applications connect to the whole cluster, identify the master, and then start to speak to it.

By default, Cloud Foundry provisions a dedicated MongoDB instance for you and provides you the IP, port, and other credentials. This doesn't include replication, and focuses on single IP connections rather than replica sets.

However when using Replica Sets, the connection pattern in your code is slightly different. You use a ReplSetConnection instead of a normal Connection, in ruby driver speak. So you need to know you're connecting to a Replica Set if the host you're connecting to isn't the master. If you only connect to the master though, you can use a traditional connection.

I was interested in maintainin full compatibility with Cloud Foundry's out-of-the-box experience, so first looked at using mongos in front of replicas. It is normally used to front a sharded setup, but using it without sharding is still a feature request.

So I set out looking at how to use a virtual IP have it track which node is the master. In this case, if one switches over, it will pick it up and move the virtual IP to follow. This would be mostly transparent to the application. The only handling in the app is the next query will fail, but quickly reconnect and its fine.

I'd already been using keepalived for HA within HAProxy, and one of the nice things it provides is the ability to define a script to run as a part of its local health checks. Using this, can have a script that just asks the local system "you the master?" and returns appropriately.

Below is the script:

#!/usr/bin/env ruby

require 'optparse'

options = { :hostname => '127.0.0.1',
            :port     => 27017,
            :username => nil,
            :password => nil }

optparser = OptionParser.new do |opt|
  opt.banner = "Usage: #{$0} [options]"
  opt.on('-H', '--hostname HOST', 'Hostname') { |o| options[:hostname] = o if o }
  opt.on('-P', '--port PORT', 'Hostname') { |o| options[:port] = o.to_i if o }
  opt.on('-u', '--username USER', 'Username') { |o| options[:username] = o if o }
  opt.on('-p', '--password PASSWORD', 'Password') {|o| options[:password] = o if o }
end

begin
  optparser.parse!
rescue => e
  $stderr.print e
  $stderr.print optparser
  exit 1
end

require 'rubygems'
require 'mongo'

conn = Mongo::Connection.new(options[:hostname], options[:port])
conn.db('admin').authenticate(options[:username], options[:password])
master_status = conn.db('admin').command({'isMaster'=>1})

if master_status['ismaster']
  exit 0
else
  exit 1
end

This won't return any output, but will exit with 0 if it is the master or 1 if it isn't.

Its requirements are simple... Ruby, RubyGems, mongo gem, and recommend the bson_ext gem too.

Then within our keepalived.conf file, it looks like this:

global_defs {
   notification_email {
     me@example.com
   }
   notification_email_from system@example.com
   smtp_server 192.168.1.1
   smtp_connect_timeout 30
}

# Define the script used to check if mongod is running
vrrp_script chk_mongod {
    script "killall -0 mongod"
    interval 2 # every two seconds
    weight 2
}

# Define the script to see if the local node is the primary
vrrp_script chk_mongo_primary {
    script "/usr/local/mongodb/bin/mongo_check_primary -u admin -p password"
    interval 2 # every two seconds
    weight 2
}

# Configuation for the virtual interface
vrrp_instance VI_1 {
    interface bond0
    state node MASTER        # SLAVE on the other nodes
    priority 101             # 100 on other nodes
    virtual_router_id 55

    authentication {
        auth_type AH
        auth_pass secret     # Set this to some secret phrase
    }

    # The virtual ip address shared between the two nodes
    virtual_ipaddress {
        192.168.1.222
    }

    # Use the script above to check if we should fail over
    track_script {
        chk_mongod
        chk_mongo_primary
    }
}

With this in place, if a server goes down, it'll switch. If mongod dies, it will switch over (since Mongo itself will recognize that), and if the current primary is switched, the mongo_check_primary will return an exit code of 1, causing it to switch over.

Another reason this might be useful in some cases is because some clients still don't support replica sets. With this method, they don't need to.

In addition to these steps, you will need to perform the normal steps to setup keepalived. These would include:

• Ensure mongod is bound to 0.0.0.0 so it'll accept any incoming connection.
• Set net.ipv4.ip_nonlocal_bind=1 in sysctl so you can use virtual IPs.

Try this out and let me know your experiences. If you're interested in this kind of seamless infrastructure automation, check out PaaS.io or drop me a line.

Recently I moved my blog over to the service I am currently working on building, PaaS.io.

Running Jekyll on PaaS.io isn't all that different from running it on other services, though I had a few other goals in mind. The way I wanted it set up was:

• Stop using rack-jekyll. Its a nice gem, however it is locked to an older version of Jekyll. And the current gem is on an even more outdated one. Currently, Cloud Foundry doesn't support pulling bundler sources from git too.
• Have a public folder for static content like CSS and images.
• Have the _site folder for generated content
• Don't have it copy the Gemfile and the config.ru into the _site folder (annoys me)
• Redirect www.invalidlogic.com to invalidlogic.com
• Low foot print

First, the Gemfile:

source :rubygems

gem 'rack-contrib', :require => 'rack/contrib/try_static'
gem 'rack-redirect'
gem 'thin'

group :development do
  gem 'jekyll'
  gem 'RedCloth'
  gem 'rdiscount'
end

The main gems being used are thin, rack-contrib (for TryStatic, note on that later), and rack-redirect (for www redirection). I also include some of the gems I use for Jekyll in the development group. That way they are available locally but not loaded when I deploy (lower footprint... and yes, it is minor).

Now, the config.ru:

require 'rubygems'
require 'bundler'
Bundler.require

use Rack::EY::Solo::DomainRedirect

use Rack::TryStatic,
    :root => "_site",
    :urls => %w[/],
    :try => ['.html', 'index.html', '/index.html']

use Rack::Static,
    :root => "public",
    :urls => %w[/]

run lambda { [404, {'Content-Type' => 'text/html'}, ['Not Found']]}

There are 4 rack components here. First, Rack::EY::Solo::DomainRedirect is the rack-redirect gem and handles the www redirection. Next, is Rack::TryStatic. It is used to access files from the _site generated content directory. It gives a couple different :try values for different ways to find the intended file. Then is the Rack::Static which gets static content from the public directory. No need to try different combinations. And last is a generic lambda that will return 404.

Next, want to avoid duplication. With things as they are, when I run jekyll it will copy the public and other items into the _site folder duplicating it. To resolve that, in our _config.yml, can add an exclude line:

exclude: [ 'public', 'Gemfile', 'Gemfile.lock', 'config.ru' ]

And with that, we are set! All of our goals are met. Commit and push to deploy! Currently I have Cloud Foundry set up with a Rack framework defined (will be sending a pull request with it soon) and also have my blog set to use Ruby 1.9.3 as well.

Soon I'll be providing some more details on PaaS.io, so stay tuned and click over to it and sign up to get access to the beta.

After spending 18 months with Android, I am now back to an iPhone and likely here to stay. While Android isn't necessarily bad, it more boils down to iPhone simply being better. After a while, I started looking at iPhone vs Android as largely a repeat of the Mac vs PC comparisons.

Mac vs PC

Just look at it. Apple is doing what they've always done. They control the hardware and the software. They have a solid, unified experience. You can pick up any iPhone, old or new, and still be at home.

Android is repeating the history of PCs. Google makes the OS, as Microsoft did with Windows. Then OEMs offer a wide variety of different hardware, with their own shitty customizations layered on top of the OS. Even worse, you have the carriers layering on their customizations and restrictions.

You can go from one Android phone to another and have it be completely foreign. Even baseline apps can have different names and a completely different look and feel, such between the "Email" vs "Mail" of vanilla Android and HTC Android.

Shelf Life

With Android, the phones have a very short shelf life. I bought a Thunderbolt from Verizon back in April, just 6 months ago. About a month after I bought it, it was no longer the hot model they were pimping. In fact, there have been 2-3 phones to come out since then that became "king of the mountain."

With iPhone, they release a new phone about once a year, and that one stays the current phone. Older phones are still pretty well supported. The iPhone 3GS is over 2 years old, still got updated to iOS 5, and likely will until iOS 6. My original Motorola Droid that is nearing 2 years old is pretty much forgotten already.

Updates with iPhone? Available right away to everyone. AT&T or Verizon, you get the update when Apple makes it generally available.

Updates with Android? Have fun. Google has to release it, your OEM has to customize it, then the carrier gets to tweak it, and decide when to finally roll it out. Google released Gingerbread back in December 2010, nearly 10 months ago. Verizon just started rolling out the update last month, then pulled it. Even then, they decide when you can upgrade with a phased roll out. And since it was pulled, looks like they seem to skip over adequate testing.

Most Android users who want recent releases end up rooting their phones and use unofficial ROMs put together by an informal group of people. Have an issue with your phone? Limited options.

Marketing

I definitely agreed with my friend Joe in his post about the Droid Bionic. Android phones are being pitched like PCs. They give a bunch of technical specs that are meaningless to consumers. It echos the "Golden Circle" idea by Simon Sinek. Apple is still doing what they do best. Verizon sells like PC manufacturers. Lacking inspiration.

False Sense of Market Share

You've probably seen the headlines of "Android sales outpacing iPhone" or "Android market share to surpass iPhone".

Overall the numbers are comparing apples to oranges. They are comparing the sales figure of a single phone to a whole group of phones. There are many different Android phones, and individually, iPhone is spanking them in sales figures. No single Android phone has a chance of elipsing the iPhone or gaining any meaningful market share, especially with their overall short shelf life. I'd be very interested to see sales figured of individual phones and see how long they really last. All the manufacturers are still struggling to get a sliver of what the iPhone is capable of.

The numbers may hold some weight for developers because it represents the size of your audience. But at the same time, Android is many phones vs iPhones entire existence is now just 5 phones. Less "your app doesn't work on my Verizon Droid Mumbojumbo" and you don't have a Droid Mumbojumbo to test on.

Falseness of Open

Many people tout Android as being open, however the actions of Google with Honeycomb's source code seem to be heading in the direction of more closed.

While Google said part of their goal was to try and unify the platform more, as Honeycomb was intended for tablets and not handsets, the ability to control a platform is difficult while also keeping it "open." I think Google is right in closing it, since in order to further the platform, they will need to have some control in order to maintain a consistent direction.

However, the "openness" usually just comes from developers. What do most of then define the "openness" as? Being able to write apps and put them on their phone without paying $99. They tout the source being open, but the truth is that isn't what they really care about. Very few Android developers likely dive into the OS code, they just want to install their own apps for free.

Personally, if I prefer one platform over the other, a $99 fee to build apps isn't going to a stopper for me. But I also don't mind paying for the tools I use in my craft if they are worth it.

Working

Most average people care more about how well the phone works. iPhone simply works better. Since software and hardware are more married, the experience is more consistent. In my own experience, apps crash less on iPhone. The phone lags less. Scrolling and browsing is more graceful. My wife is still on her original Droid for another month, and every day I see her struggle with the phone.

iPhone has better customer satisfaction ratings than Android. While the reasons aren't mentioned, I wouldn't be surprised that part of it stems from "it just works".

Apple has a strong emphasis on usability. Google and the OEMs aren't as much so. This particularly stuck me with a post I read about a guy talking to a girl who had an Android phone and an iPod Touch. In particuar, "nothing happened when I plugged it into my computer." To the average user, the simple integration is important. They don't know why the phone shows up as a "Mass Storage Device" when they plug it in.

Future

The future for Android will probable improve. The OS is maturing and hardware getting better, however the ecosystem of manufacturers and carriers will likely stay the same. The reality is that Apple is doing what Apple has always done and they've gotten really good at it.

A little late, but I have posted my slides from my talk at the East Bay Ruby Meetup in June.

Check them out and feel free to ping me if there are any questions. At the meetup, was also talking about doing a bit of a blog series on getting started with Chef and posting some of the scripts and baseline setup I have used before. Hope to start forming some simple getting started resources.

Finally posting about it a little bit late, but last week was my first week in a new position with a new company. I truly enjoyed my stay at Involver, but saw an opportunity pop up and decided to take it.

This new opportunity is to be one of the first in-house Ruby developers at Demandbase. Demandbase specializes in a B2B analytics API for reporting and logging information about visitors hitting your business's website.

That all sounds fancy, but the gist is I'll be helping to grow and form the team, scale up and expand the platform, and able to leverage both skillsets as a developer and in operations. They've been working with the awesome guys from Highgroove, utilizing Chef in a continuous deployment setup, doing geographic load balancing, and have a pretty agile and responsive process setup.

I had an amazing time at Involver and grew so much personally and professionally. I was a part of some major projects and instrumented a lot of change. Our Operations team was top notch and put out of a lot of stuff considering it was only 3 people.

Going forward, have some goodies I'll hopefully be working on and open sourcing. There are a lot of interesting things in the works and hope to post about them over time.

And by the way, Demandbase is hiring. Hit me up @krobertson or shoot me an email.

Pleased to announce Gemstache, a new service to enable teams and companies to build and distribute their own private gems with security and ease of use in mind.

Gems are an incredibly simple way to package up code and distribute it, making it available to multiple code bases or just a great way to encapsulate functionality. However, the way gems are traditionally distributed doesn't lend itself well to securing the gems and controlling access. The default "gem install" command you run doesn't provide any support for authentication against the gem source, so most sources are open to the public.

Gemstache works by giving you your own private gem source you can use. In order to download any gems from it, you must have the accompanied "gemstache" gem installed locally or on your servers. When talking to your private gem source, it interlaces to add authentication to the request as well as ensuring it is happening over HTTPS. With this, your gems are only downloaded over SSL and any interaction requires authentication.

To make managing your gems easier, the gem adds the ability to easily upload a gem to your private gem source and make it available. From your command line, just run "gem stache [your gem file]" and it is uploaded and ready for use.

Within the service, you can give users varying roles to define whether they can just download gems, are able to publish new gems, and are able to yank/delete gems.

Gemstache was born out of a need we had been facing at Involver, where some of our own internal code is enclosed in gems and used between a few of our codebases, as well as we have some gems we have forked to either add customizations or our own fixes. Traditional methods left them exposed and also more painstaking to release.

We're just about ready to launch a public beta. Visit gemstache.com and sign up to be notified.