In Valid Logic

Pleased to announce that Trunks is live! Head over to the new Trunks blog for the announcement.

We are now open to beta requests and will be sending them out here and there. I’m really wanting to progressively increase the number of users, rather than unleashing hell on it.

I’ve had everything pretty much ready to deploy since Friday, but then decided to wait until Monday, but then after doing the whole family pictures with Santa on Sunday night, was too pooped to write up the blog post and the final tweaks on the website. Plus I was going to a concert last night. Last night, finally got the last two things all done and everything was set for this morning.

So please head to the site, sign up to receive a beta invite, and enjoy!

With every new online service, the question on everyone’s mind is always "what is it going to cost?" Or perhaps these days, "how much will I get for free?" The "freemium" model has become very prevalent in online services, where you get a basic set of services for free and then pay for the really good stuff.

Trunks is a bootstrapped service on a small budget. Trunks will run off hardware I already own, in datacenter space I already have space in. No cloud wizardry here. It is cheaper for me to shift around stuff I was already running to free up a few servers than to do anything else.

My goal with Trunks is to grow it organically (sweet, got in a buzz word). As it grows, I want to ensure it is covering the costs it generates. If it grows slowly, that is fine. That would actually be better. I’d love to see the hardware gradually grow in utilization than to have explosive growth and be beyond capacity.

In my view, there are two problems with the freemium model to where it won’t fit with my goals.

First, conversion ratios are unknown. At the time you launch your service, you have no idea what your conversation ratio will be, and thus your profitability and financial stability. Good ratios are often considered to be 4% or greater. Some companies get awesome conversion ratios. But it can go very bad too. You may end up with only 2%, 1%… or 0. With my budget, I don’t have the capital to risk on a bad conversion ratio.

Second, your paid users are carrying your free users. A conversion ratio of around 5% is often considered good, but with that you have one paid user covering 19 free users. The argument for freemium is that with most online services, the cost per user is so minimal that it doesn’t matter. But with hosted source control, I’m essentially selling storage which has a much more noticeable cost. Storage is made up of capacity and throughput, and you can only have one of the other. Throughput is definitely better since otherwise you’ll get poor performance before you use all your capacity.

So where is all this leading?

Trunks will be a "premium" service. When you sign up, you will get a 30 day trial. At the end of the trial, you either upgrade to a paid plan or your repositories get locked. If your account gets locked, the repositories won’t be accessible, but will be kept around for a while in case you decide to come back.

The cost of a paid plan? The entry level plan will be $20-30/year for 250mb storage. Yes, you read that right if it sounds too low. Since my target audience is individual developers, I wanted to keep it to a reasonable price. Developers often love getting toys, but need to get spouse approval too. Since everyone using the service beyond 30 days is paying, users are paying for themselves and not themselves and 19 others. I can have a lower price while still maintaining a decent profit margin. Additionally, I sell based on space. You get 250mb, which will likely be acceptable for most. If you need more space, then that is when you upgrade.

Other features will include:

• Unlimited repositories
• Unlimited collaborators
• SSL Encryption
• Dual Remote Backups

With collaborators, one thing that will be possible is for you to collaborate with others, where you can give them commit access to one of your repositories. They don’t need a paid account either. After your trial period, if you don’t pay for an account, you will be classified as a collaborator. You can’t have your own repositories, but you can still commit to other user’s repositories that have been shared with you.

SSL Encryption is obvious: keep your communications secure. The website is entirely over HTTPS. SVN access is entirely over HTTPS. And all mercurial and git access is over SSH.

The dual backups is where data is backed up to two offsite locations. One site is backed up to with every commit, the other one is backed up to nightly, and maintains a history of repositories.

Wanted to just given a quick glance at what Trunks is looking like. I’ve been making a number of changes in the UI, making it easy to navigate and display the most pertinent information.

First up, the repository listing. I wanted this to really focus on a high number of repositories without making the screen overly cluttered. It simply lists a small box with the repository name, and its source control URL, so if you needed to quickly grab one, can easily come here and copy the URL you need to get at the code. Or click on the title and browse the latest code.

And second, we have a directory listing. Here, you see the current directory, an excerpt of the most recent checkin message for that file or directory, and the commit number (or part of the SHA hash for git) of the most recent commit. At the top, is the most recent commit for the directory. Also in the directory listing, it will highlight the commit number of files or directories that were modified in the most recent commit. If you navigate deeper, it will continue to show the most recent commit for that part of the code, letting you get down to recent changes quickly.

And finally, the commit history. This is a day-separate list of all the commits for the repository with quick links to view the commit to see individual changes, and you can also link over to view the code tree as of that commit.

All and all, it is coming along very nicely. This past weekend, I already soft launched the site to work out all the deployment kinks. Seeing as how I wrote the majority of it a year ago, functionality wise it is pretty complete and I’ve been using it for a while, but working on the lose ends to be able to launch it to a wider audience. You know, things like forgotten passwords.

Check back soon for more!

Sometimes its just fun to write an app for the sole purpose of playing around with something new.

I’d heard of Mail Hooks before, which is a service that turns an email into an HTTP POST. I’d been wanting a reason to play with XMPP, and found a way to. I’d been wanting to find a better way to test email functionality in Telligent Community/Enterprise but with live messages. Typically, this is annoying to set up since you need live email accounts, and need to set up the Windows service for them, and so on. On a dev machine, often want to do it quick and easy, and will have to regularly re-do the configuration as you switch or rebuild databases. Using HTTP POSTs for messages also wouldn’t work, because most of the developers are behind firewalls or NAT routers (like myself). I needed something else to send the message.

So what better than an SMTP to XMPP gateway, and then have a simple desktop app I fire up when I want to test messages that acts as an XMPP client. ~120 lines of ruby and more lines than that in a WPF app, and I got an easy to use solution.

The SMTP server is a very simplistic implementation, but it maintains a list of contacts that are online and allows anything@username.domain.com to be routed to them. I set it up with ejabberd and have it allow new user registration, and it auto-notifies the SMTP service user. When the SMTP service gets a notification of a new user, it automatically subscribes to their presence updates, and the client is set up to auto-accept the request.

When whenever the desktop app receives a new message, it sends it over to Telligent Community just as if our normal Windows service did.

Feel free to use the Ruby code as you wish.

Wanted to take a moment to announce a project that I’ve been progressively working on over the past year, and hope to finally open up soon.

Hosted source control is a market that has really exploded in the past 18 months, with a number of providers and options out there. I’m a huge fan of source control for all code, whether big or small, but with most of the offerings, I found something lacking. There wasn’t really a service that could meet all of my needs. After dwelling on it for a while, started to figure out what I was really looking for and decided to turn the idea into a reality.

Trunks started out as a realization of a few core difference between what was out there and what I was looking for.

• Many other services are opinionated about their source control tool, but the feature set people are looking for in a service are the same regardless of the tool. Additionally, people often use multiple tools. I use git for my ruby development, but subversion for most of my .NET development. I don’t want to have to pay for two separate services.

• Most developers are tinkerers. They work on stuff here and there and have an ever growing collection of code and projects. I believe all code should be under source control and never thrown out, but over time I’d be paying more and more with other services due to the number of repositories.

• Some of the services sell to companies more than individuals and are loaded up with other features. A lot of hosted source control services are bundled with extensive project management, focused around teams and collaboration on the team. I just wanted plain and simple source control.

This has lead to three of the driving ideals in Trunks:

• SCM agnostic. Trunks currently supports Subversion, Git, and Mercurial, with support for Bazaar also in the works.

• Unlimited private repositories. And based on focusing on individuals and their growing projects, I am still on the fence about even having public repositories (open to feedback).

• Just source control. I do plan on some very basic issue tracking (a numbered todo list?), but nothing fancy. It should just be plain simple to use, pretty to look at, stable, and reliable.

More details will be coming. I’ve had the service up and running for almost a year, but have been sitting on it to figure out what I really wanted to do with it. I recently started dusting it off and diving back into it, polishing the points I liked and removing the annoyances I found while just using it for a few months. Hopefully over the next couple of weeks, I’ll feel confident in it enough to go to a private beta and really start kicking its tires.

Keep tuned for more!

In this post, I’ll cover the specifics of the search mappers. Lets say we have a class named Book and we want to index it along with everything else Telligent Community normally indexes. I’d first create a project called BookSample and add a reference to the Telligent.Search.Mapping assembly.

Mappers are classes that inherit from the Telligent.Search.Mapping.DocumentMapper class. They have to implement two pieces, a Load() method and a Priority getter. The Load() method is used to define function blocks that are later executed to perform the mapping. The priority is to allow some overriding over the mapping process. The mappers allows support inheritance, so you could define a mapper for a Post (even though Post is an abstract class), and you could also define one for ForumPost. When a ForumPost object is mapper, both will be executed.

Additionally, you can define multiple mappers for the same type and the order of execution is set with the Priority value. All built-in mappers have a priority of 10, so if you wanted all post bodies to be indexed in lower case (for instance), you could define a mapper with a priority of 20, so it will run after the default one, remove the Body field, and then re-add it as lower case.

Our Book search mapper would look something like this:

using Telligent.Search.Mapping;

namespace BookSample
{
  public class BookMapper : DocumentMapper
  {
    public override void Load()
    {
      Map<Book>(book =>
        {
          book.ISBN.MapTo("ISBN").Key();
          book.Title.MapTo("Title");
          book.Description.MapTo("Description");
          book.Author.Name.MapTo("AuthorName");
          book.Price.ToString("0.##").MapTo("Price");
        });

      Rehydrate<Book>(isbn => Books.GetByIsbn(isbn));
    }

    public override int Priority
    {
      get { return 10; }
    }
  }
}

A few things to note:

1. The Load() method includes calls to the Map() and Rehydrate() methods, where you define an Action/Func block for the corresponding type. This may sound funky, since we have gotten some questions about why we just didn’t have something like DocumentMapper. We didn’t really want to limit a mapper definition to a single type, we wanted them to be more "scriptable", where you could clear or remove existing mappings and redefine your own, and we wanted a more fluent syntax so Map says "I’m mapping a Book" where Map says "I’m mapping an Author". It could have been done either way, we just ended up with it this way.
2. In the Map call, you are passed in a variable which is the actual object you will be mapping. The action block is called at a later time, and when it is, the Book that needs mapping will be passed in, so you can access is just as you would anywhere else in your code. To add a field onto the search document, all you need to do is use the MapTo() extension method on whatever you want to add and specify its name in the search document. So we have a property for book.Title and in the search document we want it named "Title", so we call book.Title.MapTo("Title").
3. All search documents need a Key. This is whatever the original ID was of the object. If and object is mapped and none of its mappers set a Key, it will throw an exception. This is required because if the object is later reindexed, it uses a combination of the type (Book) and the key (the ISBN value) to remove the old document from the index and add the new one. Additionally, we provide a way so if you have a search document, you can "rehydrate" it and get the original object (see later points).
4. Again, the MapTo just needs to be whatever final value you want added, so if you want to add something from another object, just access it. In this case, book.Author represents another class type. We don’t want to index it, we just want to include the author’s name. All you need to do is call book.Author.Name and then use MapTo on it. In most cases, our fields in the search document map the property name, but in this case we want to use "AuthorName" so it is a clearer it doesn’t relate to the book.
5. And for a third time, MapTo is called on the value, so it can also be manipulated. With book.Price, it is a double and we always want it indexed with two decimal places. Simply need to call the ToString() on it and format it how you wish, then call MapTo on that.
6. MapTo can be used on any type, as it is an extension method for "object" and not just common types like string, int, etc. However, if MapTo is not a string, it will call ToString() on it to convert it. This is to make the mappings a little less verbose like post.PostID.ToString().MapTo("PostID"), but it also means it will adhere to some defaults and if you want a specific format, you need to convert it yourself. Additionally, if you try to map a custom class, like the Author one mentioned earlier, it will call Author.ToString(), which if you haven’t defined in your class, the field will likely be useless to you.
7. And finally, after the call to Map, we also define Rehydrate. This takes the key we previously set in the mapping, and allows to you define a function that can take that original key and return your object. This is useful in cases where you have a search document representing the Book, but want the full object. Be aware though, it can have performance costs. If you have 20 search results and look through each one, rehydrating them to display some data, that might be 20 DB calls.

In the next post, will go over how to implement the content handler for your new type.

Although I am still on paternity leave (have 2 more weeks), I am still trying to get a few things done to stay in the swing of things. Earlier today, I saw Bruno Lopes post on Twitter enquiring about how to add your own types to the new search in Telligent Community/Telligent Enterprise 5.0.

This will be a series of posts about some of the components of the new search and some of the things you need to do to add your own objects to the index and get them back in the results. This post will focus on just the components of the new search at a high level.

In previous versions of Community Server, search had a very narrow purpose. Search Barrel and the Enterprise Search were almost entirely separate, meaning if you tried to add support for additional types, it would be specific to the form of search you were using. It had very few extensibility points for integrating, and the implementation was very specific to posts.

The new search is built more like a framework, without any specific ties to certain types of objects. It is also built to be easily extended and even replaced. It is made up of several components: mappers, content handlers, an index provider, and a search provider.

Mappers define what objects and values on those objects will be stored in the index. The index is made of a series of “search documents,” which basically outlines what object it represents (eg, a ForumPost), what is original ID was (as a string, such as PostID “1234″), and a collection of fields like Subject, Body, Author, etc.

Content handlers are simple things that are used to find content that needs to be indexed, makes a call to map it to search documents, passes it over to the index provider, and then somehow stores a setting to know that piece was already indexed.

Then the index provider and search provider implement the specifics of the search backend. Included in Telligent Community is a set of providers for Solr, though you could implement your own to work with FAST, Yahoo BOSS, or whatever else you wish. The providers mainly focus on handling search documents, so you can change out the providers and they’ll automatically work with your existing mappings and content handlers.

In the next post, I’ll cover how to write your own mapper and some of the details about how mappers work.

In the newest version of Telligent Community and Telligent Enterprise, we’ve completely rebuild the way email templates are handled to unlock a new level of possibilities and performance. I’m planning on going over several aspects of it, including highlights, usage, and extending the new templating with a series of blog posts.

Up until now, we’ve always used the build in System.Net.Mail.MailMessage support which is baked into the framework. While it has served us well, it had a couple of limitations that we had been bumping up against in trying to grow Mail Gateway’s functionality.

The new templating is based on nVelocity templates and borrows a lot from Graffiti. If you’ve ever created a Graffiti theme or written a Chalk extension, you’ll see a lot of similarity with the new templating. With the messages now, you actually generate the raw MIME messages. MIME stands for Multipurpose Internet Mail Extensions and it is the standardized format of email messages. By working with MIME directly, aided by a series of helpers, we are able to control aspects of the message in a fine grained manner.

First, in ability to control who we’re emailing independent of the message headers. With email, who the message shows in the To headers who it isn’t necessarily sent to, such as with mailing lists. Previously, Mail Gateway always sent the messages to the user, from the sender, and with the list as a special “Sender” header. This is how messages showed up in Outlook as from the user “on behalf of” the list. This type of set up didn’t work well with all client, particularly the iPhone. The iPhone would always reply to the user who sent the message, not the list.

Building on the last point, since we couldn’t set who a message is to independent of the message, Telligent Community would have to generate a message for each person it was emailing. So if a forum had 100 subscribers, when a new post is made, it would generate the email 100 times, queue 100 messages, and then send it out as 100 individual emails. With the new handling, we can actually generate the message just once and have it go to everyone in one swoop. Also, since mail servers often limit the number of recipients per message, we can take that message and configure it to allow 20 recipients per message. So instead of sending 100 messages, we could safely get it down to just 5.

Additionally, the variety of messages can be greatly expanded. It is now super easy to include an HTML and a plain text version of the body in the same message, then allow the email client to chose which one it will show based on its preferences. Or we can generate both at once, and then chose which one gets sent out after wards depending on user preference.

With the old support, we didn’t have the ability to support subsets of functionality like meeting requests. Meeting requests are basically an attachment with the ical event, and special headers indicating a response is supported/expected. With the built in functionality, Mail Gateway could read the attachment, but not handle or send it back out as a meeting request. We still haven’t added this functionality, but now the limitation from MailMessage is removed so we’re one step closer.

And finally, the old templates were limiting with what was available within messages. You only had a fixed set of tokens that could be used within the email, such as [SiteName], but it was really limiting in additional fields or field manipulation. Do you want to change the format a post date is displayed in? Want to only show a certain excerpt of the post body? Want a field that wasn’t available in the standard set? These things would often require code changes before, but now with nVelocity templates, you actually get the Post object, not just the common post tokens. Want to change the date format? Just use $post.PostDate.ToShortDateString(). Want the post author’s display name? Use $post.User.DisplayName.

Aside from new functionality that is enabled, there are several benefits to what we already use emails for.

First, email templates themselves are greatly simplified. You no longer have a gigantic XML file with all templates, they’re now template-per-file. Templates are no longer in XML, so you don’t need to worry about escaping certain characters of using
instead of
or using an amperstand (&) or other characters that need to be escaped in XML. These minor things often confused users who didn’t know about XML, or even seasoned developers who had a knee-jerk moment.

Second, when you need to do something more complicated, there are now some solid extensibility points. This makes it much easier to integrate advanced functionality in a simple way.

More advanced pieces of email generation is now simpler to test. With the extensibility, code is compartmentalized and easier to break down and test certain subsets. Additionally, we also plan on making it easier to unit test specific email templates, to ensure the template is rendered without errors and with the right content. I’ll be covered the ways to extend the templating in upcoming posts.

Finally, for portions like forum notifications, the number of messages generated is reduced which helps to improve overall performance.

In the next post, will give an overview of ways to extend the functionality of the templates.

Last week, ENTP/Active Reload open sourced their web-based Subversion browser Warehouse. I’ve been using Warehouse on my own box for about six months for my personal Subversion repositories and had been anxiously awaiting their next release, since it was supposed to add support for Git.

With it being open sourced, and on Github, I went ahead and forked it almost right away. One of the first things I wanted to do was to port some of my customizations from 1.1.6 over to the latest code. I had made a number of tweaks to make it so that Warehouse itself managed the Subversion repositories, rather than you pointing it to an existing repository. Also fixed up a few incomplete/unused parts like its dashboard, added the ability to display README files inline with the directory listing (like Github) and so on.

With the open sourcing, it looks like a lot of their support for Git was half done. It supports browsing and changesets, but has nothing for adding Git repositories, and have had a few issues with syncing changesets. I’ve fixed it up so that things basically work, though unsure about how much effort it would take to finish it. If you’re using something like Gitosis (which I am), don’t know if a lot of the management is needed. Gitosis takes care of most of creating repositories, SSH keys, and permisisons, but it is still nice to have the repositories browsable on the web and easily manageable. If collaborating with some others on a project, it is nice to create an account for them and add their SSH keys themselves. Could have Warehouse write to and commit the gitosis.config, though that seems a little funky.

Overall, I am not sure how far I am going to take my improvements on Warehouse. But in general, the code could use a lot of clean up, as the author admits. What would be really nice would be a rewrite in Merb/DataMapper, and designed to be more SCM agnostic, so it could also work with Bazaar or Mercurial.

Another thought is to have built in WebDAV support, so it isn’t dependent upon outside processes for serving the actual files. Right now, it basically assumes mod_dav_svn in Apache, likely Gitosis for git, or possibly WebDAV. But with those, taking the management of passwords and permissions takes additional steps to set up… you can’t just install and run. Though rewriting would be a big undertaking, which I’m not sure if I am up for. I tend to jump between extracurricular projects, as they tend to mound up.

If you do clone my code, be aware the installer is currently broke. My changes introduced some new configuration settings that the default installer doesn’t create. Going to be fixing that.

I was browsing some code for utilizing webservices in Ruby the other day when I stumbled upon something that made me raise my eyebrow in a WTF moment.

This library, which will remain nameless, was pretty simple in nature, but it included some extensions to core types like File, String, Fixnum, and Symbol, which made it so you could do this:

def get(obj)
  siteroot = 'mydomain.com/api'
  open(http://siteroot/obj.xml) do |res|
    ...
  end
end

I found it quite odd… creating URLs without making them strings? Ok, kind of cool to look at. But not very practical. Sure you can tack some values together easily, but how useful is that? Don’t really gain any productivity. Could easily just write it like this:

def get(obj)
  siteroot = 'mydomain.com/api'
  open("http://#{siteroot}/#{obj}.xml") do |res|
    ...
  end
end

Not that much harder, and much clearer that it is a string! The other way is far less readable, since you might get tripped up on what is a variable and what isn’t. Upon first glance, might not realize ‘obj’ is a variable and not a wannabe string. But the .xml? That isn’t a variable, that is a wannabe string.

But it made me think… should a library take the liberty to extend the core, unless that is the purpose of the library? Should a small (this was ~50 lines) library for aiding with webservices be extending the core classes? You include this library and then the extensions are in your whole app, potentially colliding with others. Since it made use of the method_missing method, what if another library you are using takes the same liberties on the same class, then you could get some odd behavior.

It is important to distinguish between a framework and a library though. ASP.NET/ASP.NET MVC, Rails, Merb are all frameworks. This one I was looking at was a library. Frameworks are more encompassing. You build a program on them, not with them… to put it visually, a library is a piece of a building, while a framework is the foundation.

I love Ruby, but one of the side effects of a dynamic language like it is that people can go overboard with the metaprogramming and can alter behavior in ways that don’t always make sense or that could interfere with each other.

Moral of the story: Simple libraries shouldn’t modify core classes. Nifty meta tricks aren’t needed.