List Express is alive again! Plus, some details on the lifeline of the NNTP/FTP servers.
Quite glad to be able to say that work has begun again on List Express, the mailing list component for Community Server. I’d originally done a proof of concept back around October of last year, but it was very crude and based on a very early version of CS (I think it was just on like v1.0 beta 1).
Beginning work on it again, we are going back and starting from scratch. The POC (proof of concept, not piece of crap) was really the first application we tried writing that used Community Server outside of the web context, and since then, we’ve made major strides to make it easier (from within CS) and to figure out how the application should do it.
With the original POC, it primarily just used the base objects from Community Server, and used all of its own stored procedures and data providers for interacting with the database. This quickly proved to be problematic as there was a lot of maintenance that had to be done to keep it up to speed will CS was still in its early stages. An additional flaw was the lack of caching, so queries went to the database every time, causing degrading performance if used in a production environment.
Since then, by building the NNTP and FTP servers for Community Server, a lot of changes have been incorporated into CS to make it easier and we’ve learned a lot of lessons as what to and what not to do. The NNTP/FTP servers were a lot smaller than LE, so taking them from concept to full implementation was a lot easier. In this time, we went through many revisions as far as getting to link directly with CS’s data providers (no dependence on own sprocs), caching (improved performance), and threading (finding how to be multi-threaded while scaling, managing memory, not dependent on blocking calls). This brought up a few new issues that we sought to improve. For one, they directly used the data providers, by passing the CS accessor classes (Forums, Users, etc) which normally handled the caching. They had to have their own classes to ge the data and manage the caching (more work). This was primarily because the accessor classes in CS v1.0 were still rather dependent on the HttpContext, which caused problems.
When building Community Server v1.1, a lot of work went into removing the direct dependency on the HttpContext and using our own CSContext instead. The CSContext could then be manually instanciated and initialized by the application and specific to the current thread for only the lifetime of the command. This allowed using something like “CSContext.Current” to be safe in a multi-threaded environment. For the NNTP/FTP servers for CS v1.1, they were updated to use the built in CS accessor classes, which simplified the code specific for those applications and still allowed full data caching.
The FTP server also saw an overhaul in terms of how it interacts with the specific application components of Community Server. For v1.0, the FTP server was built only with working with the Galleries in mind. While working on CS v1.1, we began talking about writing additonal applications, such as a download.com style “File Gallery” application and that it would be great if it could be used with the FTP server as well. While in the process of building the v1.1 release of the FTP server, the specifics of how it worked while traversing the directory structure and working with files was abstracted. We created an “FtpApplicationProvider” provider, and “FtpDirectory” and “FtpFile” objects. The core of the FTP server worked only with the FtpDirectory and FtpFile objects, while using a specific FtpApplicationProvider to manage returning and creating of these objects. So, it wants a listing of files in the current directory. The core figures out the directory, says “get files”, the FtpDirectory says “ok, we are in this provider, get some FtpFiles from it”, and the application provider selects the pictures (pretending we’re in the Galleries right now), puts them into FtpFiles and returns it. This adds a number of tiers in terms of the steps to perform data operations, but it allows the FTP server to uniquely work with multiple applications, giving the application the control over how the interaction works, all without any necessary changes to the FTP server core.
Now, getting back to List Express, after my little tangent, all of the past work from the original POC and the development of the NNTP/FTP servers will prove vital in rebuilding List Express. List Express will be built to be extremely high performance, dynamically working with multiple applications (forums, blogs, and galleries), leverage as much out of Community Server’s core as it can (without sacraficing performance, a few things will be custom), and must be fully reliable and redundant.
It’ll probably be about another week before I sit down and actually start writing code specific for LE, since one of the lessons learned from the POC was that the compexity is far greater than the NNTP/FTP servers.
For one, LE will be integrating into existing infrastructures. It has to integrate with their current email servers. Email is one thing where configurations vary greatly, in terms of the applications used, the way their topology is laid out, and the amount of trust they want to have. Some will be more wary about adding in direct event sinks into their production SMTP server, so we’ll allow them to put the email into a catch-all mailbox that LE picks up the email from. Or, LE will have its own back-end SMTP server that the emails will get relayed to. This requires a lot of sensativity to ensure that any configuration they come up with can be met in one way or another.
Second, the logic involved in the processing is far more complicated and less centralized. With NNTP/FTP, they are command driven. You sit, wait for a command, get one, go off, process it, then return and wait for the next. Carried through all the commands is a context that tells you whether they’ve logged in, what user they are, their current directory/newsgroup, etc. Your center is command processor, you always leave a central point, do work, then return there. Through each step is the information context. In LE, it is more procedural. There is no direct interaction with the user, so you get an email, need to validate the To, need to validate the sender, need to validate the post it is a reply to, then add it to the database, then send out any other emails. At each check, you need other routes such as if it is to a valid forum, or the sender isn’t authorized, etc. There is no easy central point while processing a message that you branch out from and then return to. This primarily leads to complexity in terms of organization. If you just dive in and do it, then later realize you need to add this, then add that, it gets far less structured and far more difficult to maintain and keep logical. So, you need to design out all your different work flows (lovely DFDs) and make sure you capture everything in them so that you can make the organization as clear as possible.
So, in short and for conclusion, much has been happening with the NNTP/FTP servers, and much is starting to happen with List Express. We have high hopes for LE and are putting a lot of time and energy into ensuring it meets every expectation and idea we’ve come up with.