In Valid Logic

Have been working on a new take on Papercut (details coming soon) when I noticed that there were several issues opened in Codeplex and a number of patches submitted.  I’d forgotten to enable email notifications!  So decided to sit down and felt some of them merited a new release.  Release notes below.

Download

This release includes a few bug fixes and updates, several of which were contributed patches (thanks!).

• Feature: Added support for embedded images (patch submitted by jasonpenny)
• Enhancement: Show an alert when trying to use port that is already in use (Issue #5405)
• Enhancement: Don’t show the options dialog in the taskbar (patch submitted by dukesb11)
• Fix: Updates to fix issues with multipart messages (Issue #4959, patched submitted by jasonpenny)
• Fix: Issue with async networking and not handling a disposed object (patch submitted by jasonpenny)
• Fix: Out dated URL references (Issue #5457, patch submitted by billrob458)

Tonight I realized that its now been two years since I first started working with Ruby. Back in February of 2008, I’d posted about my first experiments with Rails.  Lo and behold, its now been two years.  I do say two years with Ruby though, not Rails.  I’ve been using Merb for about the past year and a half.  Although I have started messing around with Rails 3 and really like what I see so far.

So in the past two years, what have I done?  Sadly, a lot of it is sorely behind and out dated, but take a bit of a trip down memory lane…

• The SkitchDav I originally blogged about, but now no longer use
• DM-DBSlayer. I was using it in a personal app, but the app didn’t need scale, so eventually removed it.
• DM-Paperclip.  I really need to dust this off, it needs some TLC.
• Some hacking on Warehouse, which eventually led to my ideas for Trunks.
• Branches, which was basically a ruby port of Gitosis.  I wrote it anticipating using it for Trunks so I could interact with the DB in Ruby.  I did in the first incarnation of trunks, but am now on “Branches 2.0″ which was done from scratch and super simplified so it is Trunks-only and structured to do both Git and Mercurial.  Would actually be nice to pull it out of Trunks and release it.  Damn, another project for my list.
• Tigger.  Really bad name.  When I first wrote Trunks but was unsure about launching it, tried to redo the interface as a better read-only web interface to git repos.  The app should still work… was on Merb 1.0.
• And of course, Trunks.  Merb 1.1-pre.

And I’ve also had several apps that haven’t seen the light of day and were more for my personal use or tinkering.

• For a while I was into EVE Online and wrote an app to parse and process item reports from Eve Central.  It allowed me to locate items that were selling under market rate quickly, compare prices across markets, and I could somewhat defer what items were selling most at different market hubs.  Really geeky.  Probably put several months into it.  Merb app.
• Also wrote an app to track my sales by location, by item, and comparing sales volumes based on price as I played with prices.  Merb app.
• And another for EVE Online… came up with the “Jita Mineral Index” which tracked mineral prices in the main market hub in the game.  I left it up for several months after I stopped playing, until I found that the app that processed the emails from Eve Central broke and had like 200k+ unprocessed messages in its inbox.  Merb app.
• Some time ago, was playing with different virtualization APIs and wrote an app that would provision and deploy virtual servers in KVM and in XenServer.  I’m probably too much of a nut about virtualization.  Mixture of stuff… Ruby, Nanite, Merb.
• A really basic Sinatra app to handle my domains that just have static pages.  Rather than having multiple sites, I have just one and it determines which template to render based on the host header.  Was my first Sinatra app.

And of course some other stuff on Github and Gist.

And I have some that will likely be coming soon:

• This, which I’ll likely blog about once I get a name for it and a domain.  And update the incorrect launch date in the News part, oops.
• Was actually thinking of dusting off my original SkitchDav as a Sinatra app.  What triggered me to write this.
• Also thinking of dusting off an app I prototyped a while ago involving Rack and Solr.
• Always got ideas brewing in my head… just a matter of parsing out which are really worth my free time.

All this Ruby has also helped me to grow so much more as a .NET developer.

• More languages you know the better.
• Exposure to new methodologies and mindsets, new more progressive community.
• Exposure to testing.  Testing is far easier in dynamic languages, and grasping the concepts in a dynamic language makes it easier to apply to a static language.
• MVC.  Thank god for MVC.  Unlearned all the webforms crap and re-learned plain and simple HTTP and a logical structure for your app.
• Making programming fun again.  Its nothing against .NET, but sometimes you need to break out from the familiar and learn something new to remind you why you love programming.  Before Ruby, I was stuck in what I knew and don’t really think I was coding for fun much.

I have some of my own datacenter space with a couple of servers that I use for a little bit of hosting that I do.  This weekend, I’m finishing a migration and having done this type of thing a couple of times, thought I’d share a few tips.

First of all, one thing that is very nice to do, if you have a presence in both places at the same time, is to configure IP forwarding from your old site to your new site.  While it is a best practice to set the TTL on your DNS records really low (mine are currently 60 seconds), ISPs don’t always honor the TTLs and some will be more aggressive with caching.  As a rule, its usually 8-24 hours before you can really count on old caches being expired.

To combat this, forwarding any traffic that might be going to the old addresses to the new ones can be very helpful, and fairly easy to do.

There can be several ways to do this, but can often times end up more complicated.  The pure and simple way to do it is to use an application called rinetd, which is basically an “internet redirection server”.  You simple edit its configuration file, specify the old IP address and port, and the new IP address and port.  Run it and you’re good!

I run ESXi on some of my systems, so I simply created a new virtual machine with a very basic configuration and installed Ubuntu on it.  I gave it a base IP that I wasn’t going to be forwarding, installed SSH on it (to remotely manage it), and then went to town setting up what I needed forwarded.  First, I added the IPs to be forwarded as aliases in /etc/network/interfaces:

auto eth0:0
iface eth0:0 inet static
     address 192.168.1.111
     netmask 255.255.255.0

Now, I need rinetd. On Ubuntu, it is readily available as a package, so just run sudo apt-get install rinetd and you’re set. Edit the /etc/rinetd.conf to list out how you want things forwarded:

# old ip         old port    new ip          new port
192.168.1.111    80          172.16.1.222    80

Afterwards, all you need to do is restart networking and rinetd. So run sudo /etc/init.d/networking restart and sudo /etc/init.d/rinetd restart.

Few things worth noting. rinetd binds to the address/port you specify, so if you are trying to forward port 80, you can’t have something else bound to port 80 on that IP, such as a webserver. So if you need to forward something that you might have in use, you may have to tweak things. If you need to forward SSH, it might be best to configure SSH to use another port for a little while. Second, I’d sometimes just reboot the virtual machine instead of restarting networking. I’d migrate some stuff one night and have it forward for 24 hours, then next night migrate something else and have it forwarded. It’d sometimes get a little hairy adding/removing/reconfiguring aliases a lot, so I’d just reboot.

Another plus with rinetd is that a port of it exists for Windows! A download exists on its site and it is also possible to configure it as a Windows service using ServiceEx. I haven’t tried it myself, but I did find some pretty detailed instructions about configuring it on Windows.

Was recently working on an issue that was so perplexing, that I almost had David Penton, our resident SQL expert stumped.

We had an issue with some username handling and it ended up boiling down to there we could go into SQL Server, run this query, and get a result when you’d normally expect it to not match:

select 1 where N'ß' = N'ss'

After that, I was truly stumped. I’d Googled everything I could think of any came up empty. Had I known what the German sharp S was, life would have been easier. Come to find out through some sources, that some of the SQL standards (in this case, SQL92 is referenced) call for the German sharp S is translated down to “ss” in string comparisons. So although it might seem like a bug, it is to-spec. Oracle is also reported as working this way.

To work around the behavior, one option is a workaround function for SQL Server which essentially has you convert the string to a binary column and store it. This may be outside what you can do in your application though, as in ours, we’d need to update every place that does string comparisons on the username as well as altering the ASP.NET Membership tables and stored procedures.

An alternative is to change your database collation to use one that is more binary dependent such as Latin1_General_BIN. This will treat ß and ’ss’ separately, though also be aware since it is binary, it is case insensitive and some behaviors such as sorting may also change.

A topic like this is important for any developer for a few reasons:

First, being in the US, it is easy to be ignorant of other culture’s language handling. You may work to add resource labels so your application can be translated and such, but every so often you will find behaviors related to internalization that perplex you because you’ve never heard of the German sharp S (as in this case).

Second, when its an issue in SQL and you’re in doubt, look at collation first. SQL Server has a huge list of collations, and each one does something different. Its either a behavior with the one you’re using, or the user is using a collation you haven’t tried (tested) before.

And finally… need to try harder to stump Penton. It can be done!

When implementing the collaborators feature in Trunks, ran into a bit of a roadblock with how to describe the relationships between the repository and the users.

Trunks is built on Merb and DataMapper, and the implementation involved two models: a User and a Project.  In Trunks, a repository is a Project because “repository” is a reserved attribute in DataMapper and I was running into issues early on, so I changed it to Project.

A project belongs to a user, and a user has many projects.  A user can also be a collaborator on many projects, and a project can have many collaborators.  So I already had the first relationship which was a simple 1-to-many relationship.  The problem came about with adding collaborators.  In a sense, it is a normal many-to-many, which you can do in DataMapper similar to how you would in ActiveRecord using a “has_many …, :through => …” relationship, but I was creating a relationship between models that had an existing and separate relationship.

I didn’t want the user.projects collection to return the projects a user collaborated on, because it was intended to only be the user’s own projects.  And similarly, a project only has one owner, not many.

I essentially found I needed a “has many through” with named attributes.  To do this, I had to actually create the proxy class that goes between the user and the project model, rather than let DataMapper create it automatically.  That way, I can control the attribute names on the proxy class so that “users” and “projects” on the two models won’t get munged.

The result looked like this:

class User
  include DataMapper::Resource
  ...
  has n, :collaborations
  has n, :collab_projects, :model => 'Project', :child_key => [:id], :parent_key => [:user_id], :through => :collaborations
end

class Project
  include DataMapper::Resource
  ...
  has n, :collaborations
  has n, :collab_users, :model => 'User', :child_key => [:id], :parent_key => [:project_id], :through => :collaborations
end

class Collaboration
  include DataMapper::Resource

  property :id, Serial

  belongs_to :collab_user, :model => 'User', :child_key => [:user_id]
  belongs_to :collab_project, :model => 'Project', :child_key => [:project_id]
end

Having a “collab_users” on the Collaboration class allowed me to have the project model to get an attribute name “collab_users” instead of trying to use “users”. And on user, I could have “collab_projects” instead of mixing things up with the existing “projects” attribute.

When creating the relationships though, I had to be careful to spell out the actual parent and child fields rather than let them be automatic, otherwise it’ll run into issues where it’ll try to create ‘collab_user_id’ and ‘collab_project_id’ and inserts will fail because those columns aren’t being set.

Take the relationship definition on the user model:

	has n, :collab_projects, :model => 'Project', :child_key => [:id], :parent_key => [:user_id], :through => :collaborations

I am essentially saying create an attribute named ‘collab_projects’ through the ‘collaborations’ relationship. ‘collab_projects’ will be of type Project. The child key on Collaboration is ‘user_id’ and the primary key on User is ‘id’. Here, the fields I specify are betwen User and Collaboration, not on Project.

Then on Collaboration, I have:

	belongs_to :collab_user, :model => 'User', :child_key => [:user_id]

Here I tell it that the attribute on Collaboration will be collab_user, but the model it is linking to is User. Basically saying I belong to user, but don’t create an attribute named ‘user’. I set the child key, which is the column on Collaboration, to be ‘user_id’. I want to do that else it will try to create it as ‘collab_user_id’.

Confused enough? All boils down to having two separate relationships between two models and having control over what the attributes are named. After all this, I end up with these calls:

  user.projects        # the user's own projects
  user.collab_projects # the projects the user is a collaborator on

  project.user         # the owner of the project
  project.collab_users # the users who are collaborators on the project

Pleased to announce that Trunks is live! Head over to the new Trunks blog for the announcement.

We are now open to beta requests and will be sending them out here and there. I’m really wanting to progressively increase the number of users, rather than unleashing hell on it.

I’ve had everything pretty much ready to deploy since Friday, but then decided to wait until Monday, but then after doing the whole family pictures with Santa on Sunday night, was too pooped to write up the blog post and the final tweaks on the website. Plus I was going to a concert last night. Last night, finally got the last two things all done and everything was set for this morning.

So please head to the site, sign up to receive a beta invite, and enjoy!

With every new online service, the question on everyone’s mind is always "what is it going to cost?" Or perhaps these days, "how much will I get for free?" The "freemium" model has become very prevalent in online services, where you get a basic set of services for free and then pay for the really good stuff.

Trunks is a bootstrapped service on a small budget. Trunks will run off hardware I already own, in datacenter space I already have space in. No cloud wizardry here. It is cheaper for me to shift around stuff I was already running to free up a few servers than to do anything else.

My goal with Trunks is to grow it organically (sweet, got in a buzz word). As it grows, I want to ensure it is covering the costs it generates. If it grows slowly, that is fine. That would actually be better. I’d love to see the hardware gradually grow in utilization than to have explosive growth and be beyond capacity.

In my view, there are two problems with the freemium model to where it won’t fit with my goals.

First, conversion ratios are unknown. At the time you launch your service, you have no idea what your conversation ratio will be, and thus your profitability and financial stability. Good ratios are often considered to be 4% or greater. Some companies get awesome conversion ratios. But it can go very bad too. You may end up with only 2%, 1%… or 0. With my budget, I don’t have the capital to risk on a bad conversion ratio.

Second, your paid users are carrying your free users. A conversion ratio of around 5% is often considered good, but with that you have one paid user covering 19 free users. The argument for freemium is that with most online services, the cost per user is so minimal that it doesn’t matter. But with hosted source control, I’m essentially selling storage which has a much more noticeable cost. Storage is made up of capacity and throughput, and you can only have one of the other. Throughput is definitely better since otherwise you’ll get poor performance before you use all your capacity.

So where is all this leading?

Trunks will be a "premium" service. When you sign up, you will get a 30 day trial. At the end of the trial, you either upgrade to a paid plan or your repositories get locked. If your account gets locked, the repositories won’t be accessible, but will be kept around for a while in case you decide to come back.

The cost of a paid plan? The entry level plan will be $20-30/year for 250mb storage. Yes, you read that right if it sounds too low. Since my target audience is individual developers, I wanted to keep it to a reasonable price. Developers often love getting toys, but need to get spouse approval too. Since everyone using the service beyond 30 days is paying, users are paying for themselves and not themselves and 19 others. I can have a lower price while still maintaining a decent profit margin. Additionally, I sell based on space. You get 250mb, which will likely be acceptable for most. If you need more space, then that is when you upgrade.

Other features will include:

• Unlimited repositories
• Unlimited collaborators
• SSL Encryption
• Dual Remote Backups

With collaborators, one thing that will be possible is for you to collaborate with others, where you can give them commit access to one of your repositories. They don’t need a paid account either. After your trial period, if you don’t pay for an account, you will be classified as a collaborator. You can’t have your own repositories, but you can still commit to other user’s repositories that have been shared with you.

SSL Encryption is obvious: keep your communications secure. The website is entirely over HTTPS. SVN access is entirely over HTTPS. And all mercurial and git access is over SSH.

The dual backups is where data is backed up to two offsite locations. One site is backed up to with every commit, the other one is backed up to nightly, and maintains a history of repositories.

Wanted to just given a quick glance at what Trunks is looking like. I’ve been making a number of changes in the UI, making it easy to navigate and display the most pertinent information.

First up, the repository listing. I wanted this to really focus on a high number of repositories without making the screen overly cluttered. It simply lists a small box with the repository name, and its source control URL, so if you needed to quickly grab one, can easily come here and copy the URL you need to get at the code. Or click on the title and browse the latest code.

And second, we have a directory listing. Here, you see the current directory, an excerpt of the most recent checkin message for that file or directory, and the commit number (or part of the SHA hash for git) of the most recent commit. At the top, is the most recent commit for the directory. Also in the directory listing, it will highlight the commit number of files or directories that were modified in the most recent commit. If you navigate deeper, it will continue to show the most recent commit for that part of the code, letting you get down to recent changes quickly.

And finally, the commit history. This is a day-separate list of all the commits for the repository with quick links to view the commit to see individual changes, and you can also link over to view the code tree as of that commit.

All and all, it is coming along very nicely. This past weekend, I already soft launched the site to work out all the deployment kinks. Seeing as how I wrote the majority of it a year ago, functionality wise it is pretty complete and I’ve been using it for a while, but working on the lose ends to be able to launch it to a wider audience. You know, things like forgotten passwords.

Check back soon for more!

Sometimes its just fun to write an app for the sole purpose of playing around with something new.

I’d heard of Mail Hooks before, which is a service that turns an email into an HTTP POST. I’d been wanting a reason to play with XMPP, and found a way to. I’d been wanting to find a better way to test email functionality in Telligent Community/Enterprise but with live messages. Typically, this is annoying to set up since you need live email accounts, and need to set up the Windows service for them, and so on. On a dev machine, often want to do it quick and easy, and will have to regularly re-do the configuration as you switch or rebuild databases. Using HTTP POSTs for messages also wouldn’t work, because most of the developers are behind firewalls or NAT routers (like myself). I needed something else to send the message.

So what better than an SMTP to XMPP gateway, and then have a simple desktop app I fire up when I want to test messages that acts as an XMPP client. ~120 lines of ruby and more lines than that in a WPF app, and I got an easy to use solution.

The SMTP server is a very simplistic implementation, but it maintains a list of contacts that are online and allows anything@username.domain.com to be routed to them. I set it up with ejabberd and have it allow new user registration, and it auto-notifies the SMTP service user. When the SMTP service gets a notification of a new user, it automatically subscribes to their presence updates, and the client is set up to auto-accept the request.

When whenever the desktop app receives a new message, it sends it over to Telligent Community just as if our normal Windows service did.

Feel free to use the Ruby code as you wish.

Wanted to take a moment to announce a project that I’ve been progressively working on over the past year, and hope to finally open up soon.

Hosted source control is a market that has really exploded in the past 18 months, with a number of providers and options out there. I’m a huge fan of source control for all code, whether big or small, but with most of the offerings, I found something lacking. There wasn’t really a service that could meet all of my needs. After dwelling on it for a while, started to figure out what I was really looking for and decided to turn the idea into a reality.

Trunks started out as a realization of a few core difference between what was out there and what I was looking for.

• Many other services are opinionated about their source control tool, but the feature set people are looking for in a service are the same regardless of the tool. Additionally, people often use multiple tools. I use git for my ruby development, but subversion for most of my .NET development. I don’t want to have to pay for two separate services.

• Most developers are tinkerers. They work on stuff here and there and have an ever growing collection of code and projects. I believe all code should be under source control and never thrown out, but over time I’d be paying more and more with other services due to the number of repositories.

• Some of the services sell to companies more than individuals and are loaded up with other features. A lot of hosted source control services are bundled with extensive project management, focused around teams and collaboration on the team. I just wanted plain and simple source control.

This has lead to three of the driving ideals in Trunks:

• SCM agnostic. Trunks currently supports Subversion, Git, and Mercurial, with support for Bazaar also in the works.

• Unlimited private repositories. And based on focusing on individuals and their growing projects, I am still on the fence about even having public repositories (open to feedback).

• Just source control. I do plan on some very basic issue tracking (a numbered todo list?), but nothing fancy. It should just be plain simple to use, pretty to look at, stable, and reliable.

More details will be coming. I’ve had the service up and running for almost a year, but have been sitting on it to figure out what I really wanted to do with it. I recently started dusting it off and diving back into it, polishing the points I liked and removing the annoyances I found while just using it for a few months. Hopefully over the next couple of weeks, I’ll feel confident in it enough to go to a private beta and really start kicking its tires.

Keep tuned for more!