In Valid Logic

Endlessly expanding technology

Deploying Ubuntu 12.04 on XenServer Made Easy

To follow up on my previous post about disk errors with Ubuntu 12.04 on XenServer, I wanted to cover the process I’ve put in place for provisioning Ubuntu VMs.

With, I have a mixture between systems deployed on bare metal and virtualized. With the virtualized systems, I set out to make the provisioning as easy as if I was using an IaaS provider, while still giving me fine control over sizing and placement. Some of the decisions there are enough for another post.

Ubuntu 12.04 is the new hotness and I’d been anxiously awaiting it, with plans to progressively roll it out throughout

With XenServer, it provides an easy template for provisioning Ubuntu Lucid 10.04 VMs and even for some newer releases. However, the way XenSever provisions Ubuntu is to essentially netboot it and install it from a remote source. Because of that, we can’t simply use one of those templates buts install a newer release.

Fortunately, the template itself is very simple, and which release it installs is just a configuration parameter.

To create a new Ubuntu 12.04 template, simply log into the XenServer console and runt he following commands. It will clone the Lucid template and then change the parameter for the release to install from lucid to precise.

$ TEMPLATE_UUID=`xe template-list \
        name-label="Ubuntu Lucid Lynx 10.04 (64-bit)" params=uuid --minimal`
$ NEW_TEMPLATE_UUID=`xe vm-clone uuid=$TEMPLATE_UUID \
        new-name-label="Ubuntu Precise (64-bit)"`
$ xe template-param-set other-config:default_template=true \
        other-config:debian-release=precise uuid=$NEW_TEMPLATE_UUID

Now you will find a “Ubuntu Precise (64-bit)” option in the template list of XenCenter.

XenCenter template list

Now, we can actually provision a new box. Next, one interesting discovery was that you can pass in a kickstart script in the boot parameters options for the new VM.

Kickstart allows you to do a scripted installation, automated pretty much every aspect of the system. Instead of picking a bunch of options, it allows for an easy, repeatable process that basically leaves the machine completely ready to go.

To use a kickstart script, make a script available over HTTP somewhere on your LAN or on the public internet. By the time it is grabbed, the machine will have an IP, DNS, and all. Then, simply add ks=http://url/to/your/kickstart to the beginning part of “advanced OS boot parameters” option when selecting the installation media.

XenCenter media selection

Below is a cleaned version of the kickstart script I used on my Ubuntu VMs. The main things of note:

  • Sets up my partition table
  • Configures base system with ubuntu-minimal as well as some common packages like openssh-server, curl, wget, and screen.
  • Disables the creation of the ubuntu user (I create a normal every day user through chef)
  • Configures the fstab with barrier=0 as mentioned before
  • Disables /bin/sh from pointing to /bin/dash (personal preference)
  • Updates apt sources
  • Installs xenstore-utils
  • Downloads some auto-configuration scripts
  • Installs XenTools
  • Installs the Ubuntu virtual kernel and removes the generic one
  • Cleans up apt caches
  • Shuts down the VM.
lang en_US
langsupport en_US
keyboard us
timezone America/Los_Angeles
url --url

rootpw pa$$word   # you should replace, and use --iscrypted
auth --useshadow --enablemd5
user --disabled

bootloader --location=mbr
zerombr yes
clearpart --all --initlabel
part /boot --fstype=ext2 --size=64
part swap --size=1024
part / --fstype=ext4 --size=1 --grow

network --device=eth0 --bootproto=static --ip= --netmask= \
        --nameserver= --gateway=
firewall --disabled



# update fstab for the root partition
perl -pi -e 's/(errors=remount-ro)/noatime,nodiratime,$1,barrier=0/' /etc/fstab

# point sh to bash instead of dash
rm /bin/sh
ln -s /bin/bash /bin/sh

# add normal apt source list
cat <<'EOP'
deb precise main restricted universe
deb precise-security main restricted universe
deb precise-updates main restricted universe
) > /etc/apt/sources.list
apt-get update
apt-get upgrade -y

# install some additional packages
apt-get install -y xenstore-utils

# set up xenserver automation scripts
curl $AUTOMATER_REPO/master/usr/sbin/xe-set-hostname > /usr/sbin/xe-set-hostname
curl $AUTOMATER_REPO/master/usr/sbin/xe-set-network > /usr/sbin/xe-set-network
curl $AUTOMATER_REPO/master/usr/sbin/generate-sshd-keys > /usr/sbin/generate-sshd-keys
curl $AUTOMATER_REPO/master/etc/init/xe-automator.conf > /etc/init/xe-automator.conf
chmod a+x /usr/sbin/xe-set-hostname
chmod a+x /usr/sbin/xe-set-network
chmod a+x /usr/sbin/generate-sshd-keys

# setup locales
locale-gen en_US.UTF-8
update-locale LANG="en_US.UTF-8"
echo 'LANG=en_US.UTF-8' >> /etc/environment
echo 'LC_ALL=en_US.UTF-8' >> /etc/environment

# install xe tools
cd /tmp
wget http://some/url/to/xe-guest-utilities_6.0.0-743_amd64.deb
dpkg -i xe-guest-utilities_6.0.0-743_amd64.deb

# install paravirt kernel image
apt-get install -f -y linux-virtual
dpkg -l | grep generic | grep linux | awk '{print $2}' | xargs apt-get remove -y

# clean up some stuff
rm -f /etc/ssh/ssh_host_*
rm -f /var/cache/apt/archives/*.deb
rm -f /var/cache/apt/*cache.bin
rm -f /var/lib/apt/lists/*_Packages

It is important that it shuts down at the end. My goal was to have it be all inclusive, and that means setting up the virtual kernel. However, it can’t successfully reboot, because we need to update some PV boot options before it is ready to go.

Also, some packages must be installed in the post steps. The default install sources for Ubuntu don’t always have all packages available, and I found it best to do the kernel last.

When you first bring up the new VM in XenServer, you may need to enter in a few details if you don’t have DHCP running. It will self configure by default, but I normally opt to manually configure it to ensure the template gets a certain IP just to avoid any future possible collision.

After the bootstrapping is done and the VM is then off, log into console on the XenServer host itself and run the following snippet:

$ VMNAME=precise-20120501
$ UUID=`xe vm-list name-label="$VMNAME" params=uuid --minimal`
$ EDITOR=cat xe-edit-bootloader -n "$VMNAME" -p 1 \
        -f /grub/grub.cfg > /tmp/$VMNAME-grub
$ KERNEL=`grep vmlinuz /tmp/$VMNAME-grub | grep virtual |
        grep -v recovery | awk '{print $2}'`
$ ROOT=`grep vmlinuz /tmp/$VMNAME-grub | grep virtual |
        grep -v recovery | awk '{print $3}'`
$ RAMDISK=`grep initrd /tmp/$VMNAME-grub | head -1 | awk '{print $2}'`
$ xe vm-param-set uuid=$UUID PV-bootloader-args="--kernel=$KERNEL --ramdisk=$RAMDISK"
$ xe vm-param-set uuid=$UUID PV-args="$ROOT ro quiet console=hvc0"

Before you run it, of course set the VMNAME to the name of your VM. You might also want to check some of the values as you go (echo $UUID). You can use the xe-edit-bootloader command to view the grub configuration within the VM, but need to set the PV settings outside the guest. By faking the EDITOR to cat, you can export the grub file to a local file, then use some grep+awk to get the necessary pieces and finally set the PV settings correctly.

At this point, the machine is ready to be booted, converted into a template, or simply cloned. I personally like to leave my templates as never-been-used.

One of the main benefits of an IaaS setup like OpenStack or CloudStack is the self orientating of the VMs. Normally with a template, it keeps the template’s setting for its hostname and network configuration. However, I found some example scripts on Github for passing network information into a VM through the VM’s “xenstore” options. That way on boot, the VM can read in the settings it needs and update itself. I did some work to update the scripts to work with Precise, to more thoroughly update the hostname and DNS, as well as to regenerate the ssh host keys (since the kickstart did delete them).

As an example, this would be how to clone the template to a VM, set the params, and boot it:

$ UUID=`xe vm-install template=precise-20120501 new-name-label=builder`
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/ip=
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/gw=
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/nm=
$ xe vm-param-set uuid=$UUID xenstore-data:vm-data/ns=''
$ xe vm-start uuid=$UUID

That is cool and all, but I don’t want to log into the XenServer console each time I want to setup a VM. Luckily, found a plugin for Chef’s knife utility that adds XenServer provisioning. I forked it and added setting xenstore network parameters to it.

Now, provisioning a new VM, configuring it, and bootstrapping chef on it is just a matter of one call:

$ knife xenserver vm create --vm-template precise-20120501 \
                            -x root --keep-template-networks \
                            -r "role[foo]" -E production \
                            --vm-ip --vm-name foobar

The command may be a little long, but it is all encapsulated in a single command. I simply have an internal README of sorts with the command pre-prepared for various roles.

If you have any questions, please leave a comment and ask. I’d be glad to help and always looking to improve my process as well.

Tuesday, May 01, 2012

blog comments powered by Disqus