In Valid Logic

After finishing my lengthy post, I noticed that Scott posted the v1.1 RC1 release of Community Server.  Go and check it out!  A great number of improvements have been made and is truly a excellent follow up to v1.0.

It is also worth mentioning that early next week, people should be expecting the v1.1 RC1 releases of the NNTP Server and the FTP Server, as well as the v1.1 release of the SecurityModules.

Quite glad to be able to say that work has begun again on List Express, the mailing list component for Community Server.  I’d originally done a proof of concept back around October of last year, but it was very crude and based on a very early version of CS (I think it was just on like v1.0 beta 1).

Beginning work on it again, we are going back and starting from scratch.  The POC (proof of concept, not piece of crap) was really the first application we tried writing that used Community Server outside of the web context, and since then, we’ve made major strides to make it easier (from within CS) and to figure out how the application should do it.

With the original POC, it primarily just used the base objects from Community Server, and used all of its own stored procedures and data providers for interacting with the database.  This quickly proved to be problematic as there was a lot of maintenance that had to be done to keep it up to speed will CS was still in its early stages.  An additional flaw was the lack of caching, so queries went to the database every time, causing degrading performance if used in a production environment.

Since then, by building the NNTP and FTP servers for Community Server, a lot of changes have been incorporated into CS to make it easier and we’ve learned a lot of lessons as what to and what not to do.  The NNTP/FTP servers were a lot smaller than LE, so taking them from concept to full implementation was a lot easier.  In this time, we went through many revisions as far as getting to link directly with CS’s data providers (no dependence on own sprocs), caching (improved performance), and threading (finding how to be multi-threaded while scaling, managing memory, not dependent on blocking calls).  This brought up a few new issues that we sought to improve.  For one, they directly used the data providers, by passing the CS accessor classes (Forums, Users, etc) which normally handled the caching.  They had to have their own classes to ge the data and manage the caching (more work).  This was primarily because the accessor classes in CS v1.0 were still rather dependent on the HttpContext, which caused problems.

When building Community Server v1.1, a lot of work went into removing the direct dependency on the HttpContext and using our own CSContext instead.  The CSContext could then be manually instanciated and initialized by the application and specific to the current thread for only the lifetime of the command.  This allowed using something like “CSContext.Current” to be safe in a multi-threaded environment.  For the NNTP/FTP servers for CS v1.1, they were updated to use the built in CS accessor classes, which simplified the code specific for those applications and still allowed full data caching.

The FTP server also saw an overhaul in terms of how it interacts with the specific application components of Community Server.  For v1.0, the FTP server was built only with working with the Galleries in mind.  While working on CS v1.1, we began talking about writing additonal applications, such as a download.com style “File Gallery” application and that it would be great if it could be used with the FTP server as well.  While in the process of building the v1.1 release of the FTP server, the specifics of how it worked while traversing the directory structure and working with files was abstracted.  We created an “FtpApplicationProvider” provider, and “FtpDirectory” and “FtpFile” objects.  The core of the FTP server worked only with the FtpDirectory and FtpFile objects, while using a specific FtpApplicationProvider to manage returning and creating of these objects.  So, it wants a listing of files in the current directory.  The core figures out the directory, says “get files”, the FtpDirectory says “ok, we are in this provider, get some FtpFiles from it”, and the application provider selects the pictures (pretending we’re in the Galleries right now), puts them into FtpFiles and returns it.  This adds a number of tiers in terms of the steps to perform data operations, but it allows the FTP server to uniquely work with multiple applications, giving the application the control over how the interaction works, all without any necessary changes to the FTP server core.

Now, getting back to List Express, after my little tangent, all of the past work from the original POC and the development of the NNTP/FTP servers will prove vital in rebuilding List Express.  List Express will be built to be extremely high performance, dynamically working with multiple applications (forums, blogs, and galleries), leverage as much out of Community Server’s core as it can (without sacraficing performance, a few things will be custom), and must be fully reliable and redundant.

It’ll probably be about another week before I sit down and actually start writing code specific for LE, since one of the lessons learned from the POC was that the compexity is far greater than the NNTP/FTP servers.

For one, LE will be integrating into existing infrastructures.  It has to integrate with their current email servers.  Email is one thing where configurations vary greatly, in terms of the applications used, the way their topology is laid out, and the amount of trust they want to have.  Some will be more wary about adding in direct event sinks into their production SMTP server, so we’ll allow them to put the email into a catch-all mailbox that LE picks up the email from.  Or, LE will have its own back-end SMTP server that the emails will get relayed to.  This requires a lot of sensativity to ensure that any configuration they come up with can be met in one way or another.

Second, the logic involved in the processing is far more complicated and less centralized.  With NNTP/FTP, they are command driven.  You sit, wait for a command, get one, go off, process it, then return and wait for the next.  Carried through all the commands is a context that tells you whether they’ve logged in, what user they are, their current directory/newsgroup, etc.  Your center is command processor, you always leave a central point, do work, then return there.  Through each step is the information context.  In LE, it is more procedural.  There is no direct interaction with the user, so you get an email, need to validate the To, need to validate the sender, need to validate the post it is a reply to, then add it to the database, then send out any other emails.  At each check, you need other routes such as if it is to a valid forum, or the sender isn’t authorized, etc.  There is no easy central point while processing a message that you branch out from and then return to.  This primarily leads to complexity in terms of organization.  If you just dive in and do it, then later realize you need to add this, then add that, it gets far less structured and far more difficult to maintain and keep logical.  So, you need to design out all your different work flows (lovely DFDs) and make sure you capture everything in them so that you can make the organization as clear as possible.

So, in short and for conclusion, much has been happening with the NNTP/FTP servers, and much is starting to happen with List Express.  We have high hopes for LE and are putting a lot of time and energy into ensuring it meets every expectation and idea we’ve come up with.

Ever want to torture test yourself?  Write some code to resolve a path.  The biggest pain is handling of “..” and “.” and intermixing them in ugly requests.

I’ve been working on updating the Community Server FTP Server to v1.1 and one of the weaknesses I’d found with the previous beta was some of its directory handling.  For v1.1, the server is undergoing some major overhauling (mainly to make it pluggable to work with other apps that might use files/attachments in the future).

I thought I had everything working good, until I decided to try some odder requests.  Like, if in /, typing “cd photos/sample/../sample/../../photos/” should result in changing into /photos/.  You would think that it should be simple, but no.  Had it scanning and counting occurrances of “..”, but it was looking only for consequtive ones, and then removing them all.  So it would count 2 occurrances (starts from the end back), and them remove the next two references.  It wasn’t paying attention to the fact that one of the instances it is removing might be another “..”.  So had to change it to check them while removing and not removing too much or too little.

Then I decided to start testing the “.” current directory item.  From /photos/, typing “cd ./../photos/sample/.././../” should result in /.  But again, nope.  When removing instances of “..”, it was treating “.” as a valid entry it should remove and count as occurance, when it should remove it and not count it as an occurance.  Then you get into cases where you index gets out of range and you throw an exception before it has processed all of it.  Oy.

In short, it has been a painful past two hours.  I finally got it working and all of my test cases are passing.  Yay.  I’ve fired up the debugger so many times that Visual Studio has frozen on me twice.  Just thought I’d share some excitement from my day.

It is official now.  Kevin Harder and Dan Bartels are the two newest members to the Telligent family.  They have been working on Community Server as a part of the “commit team,” where special community members are given access to our source control and bug tracker to partake in the development.  And now, they will be joining the fulltime development team who are luck enough to be paid to do what they love.

I give them my own personal congratulations and welcome them onboard.  I look forward to working with them and seeing where we all can take Community Server.  And perhaps some of the Telligent gang can break them into the other “CS” around here…

T-minus 12 hours until I am going to see the most anticipated movie of the summer, at least for me.  Didn’t want to go to a midnight showing, since that would be too crowded for me, and the morning ones are usually not very crowded, so it works for me.

Didn’t plan on it, but ended up having a bit of a late night hacking a few things into CS Galleries for v1.1.

Started out as going through some of the pictures users had emailed me that failed to be added.  Few of them were things I’d already fixed, caused by cameras that support adding the picture’s date/time, but still add the tag with all zeros (which is below .NET’s DateTime.MinValue).  That was already fixed early on post-1.0.

Then this other one was more tricky.  It was a picture that was failing the check to see if it was a valid image, even though Windows and everything else could still open it.  The code CS uses to check images without using System.Drawing is a class I ported from java (called ImageInfo).  In digging through it, I found the APP0 marker (typically has the identifier, version, DPIs, and embedded thumbnail size), which is normally 14 bytes, was only 8.  At first, I figured it was a cheaper camera that wasn’t good with standards, but the EXIF info said it was a Canon PowerShot.  So I dug deeper.  Then in going through the JPEG spec, I found that 8 could be valid if it didn’t have a DPI specified.

Then decided to dive into adding support reading the EXIF comments directly into the picture’s description.  Then, I found that in Windows XP, when you add the title/subject/comments (by right clicking and going to Properties on the picture), that doesn’t go into the jpeg comments, but the EXIF data.  So, I added support to the EXIF reader to get the Windows title/subject/keyword/author/comments tags.  Also, if you don’t specify a title for the entry, it will first look for the Windows title, then the Windows subject, and then settle on the picture’s filename.  For the description, if you didn’t enter one, it’ll first look for JPEG comments, and then settle on the Windows comments.

Have had a couple of users ask for reading comments from the picture, and now v1.1 will do that.

Now, it is 2am, and it is too late to watch Deadliest Catch.  It’ll have to wait until tomorrow.

Phil Haack has recently announced a fork of .Text called Subtext.  I wish him the best of luck, but found a some inaccuracies in his announcing post.

“There is a non-commercial license available, but it requires displaying the telligent logo and restricts usage to non-commercial purposes.”

That is only a half truth.  With the non-commercial license, you are required to display the logo somewhere on the page, but you are not restricted to non-commercial purposes.  Simply look at the second paragraph of the license:

“Subject to the restrictions below, you may use the Software for any commercial or non-commercial purpose.”

There are restrictions, but they basically consist of the logo being shown and not taking it and calling your own.  None of them have anything to do with restricting a business from utilizing the software commercially.

Secondly, his opinion of the targeted market for CS is a little off.

“By virtue of it going commercial, it’s being targeted to a different market than your average hobbyist and blogger.”

It can be difficult to write software that targets both big and small implementations.  Look at .Text.  Plenty of people had a difficult time getting it installed, regardless of large/small site.  Out of the box, CS 1.0 does lean towards a larger, multi-user implementation, but it can be easily changed over to work for single users.  CS 1.1 will be even easier, as many of the items that raised issues have been worked out.  All you will need to do is create a default blog, drop in a new url config file, and edit the web.config.  Might even be able to find a way to do all of that for you.

CS can handle small sites (like mine) all the way to larger ones, like weblogs.asp.net.  Much thought and effort is going to be going into getting it easily configurable for either.  CS1.1 has been primarily geared towards bug fixes (for performance and stability), but CS1.2 is the one to watch for some new mind blowing things (both glitsy “wow, that’s cool” down to kick ass core mechanics).

I do wish Phil the best of luck and will probably watch his progress, but naturally, my loyalties are with the software I write, and you may tend to correct someone when they are a little wrong.