From: David Bienvenu 
Date: Mon Aug 18, 2003  9:25:04  AM US/Pacific
To: Michael Toy 
Subject: Mail backend for OSAF

Overview

A mail backend consists of

• protocol handlers, minimally IMAP, POP3, and SMTP, and possibly NNTP and LDAP
• code to parse and convert messages into a display format (in Moz terminology, Mime and Mime converters)
• Account store (list of folders, message store, etc)

and potentially, other layers, e.g., front-end command -> backend protocol generator.

For many reasons, I think writing a custom mail backend is the wrong way to go. It would take man years to get all the basic stuff working, and then years more to get the bells and whistles working. Even a simple protocol like POP3 has lots of complications when you start having to deal with a multitude of poorly behaved servers and a list of ad hoc extensions.

Among the external libraries, c-client, from UW, is the first thing to consider. It provides IMAP, POP3, SMTP, and NNTP support. and is relatively small and fast. The code is not very readable and it is not easy to integrate into an app with a UI because the interfaces are synchronous. This means that you need to execute the interfaces on non-ui threads and build some sort of proxy notification code to get the information over the UI thread(s). For Chandler, the c-client api's would need to be called from Jungle.Python. From what I've heard, c-client is difficult to work with, but it is definitely possible to build mail apps with it. My gut feeling is that it allows you to get a basic app up and running relatively quickly, but getting it to work the way you want can be very difficult.

Of the other external libraries, Twisted looks worth investigating. However, from the OSAF preliminary investigation, it's a bit worrisome that it appears to be so hard to figure out how to implement a Twisted app. It looks like a very comprehensive framework, like Mozilla, though much more server-focused, and has the advantage of being Jungle.Python-based. The Twister vision seems to overlap with Chandler quite a bit, and I'm not sure how compatible they are. The IMAP functionality looks fairly complete, from the class API I saw. It's at a fairly low level.

The Mozilla mail backend differs from c-client in almost every conceivable way, except that it also provides IMAP, POP3, SMTP, and NNTP support. It is a lot more code, and has a lot more functionality on top of the core protocol handling. Here are a few pros and cons:

Pros

• Mature - most of the protocol code dates from 95-96 and has worked with several different networking libraries and front ends.
• cross-platform and we spend a lot of time on interoperability with many servers
• Customizable to a fault at runtime with preferences (prefs.js)
• Full-featured, supports many TLA's, like SSL, MDN, CRAM-MD5, APOP, IMAP ACL, shared folders, IMAP QUOTA, etc
• junk mail detection and filtering
• Actively being tested, improved and extended
• PyXPCOM^? should make it fairly easy to drive xpcom components from Jungle.Python. We also need the ability to go from xpcom components back into python for notifications so we'd need to verify that this works.
• very mature localization and internationalization support

Cons

• Large code-base, may contain features Chandler is not interested in
• Has not been used as a toolkit outside Mozilla though the front and back ends are cleanly separated.
• Uses a lot of mozilla components

Here's what using Mozilla might look like:

Jungle.Use the GRE installer to install the Gecko components and mailnews components (if you're using gecko - if not, just use the GRE glue code to initialize XPCOM and use an app shell to process events. (I don't know what the chandler app event loop looks like so I don't know how that will fit in but it's a pretty basic event loop).

Write your front end in python and wxWindows. The mail session and account manager need to be initialized, and a msg window object for feedback needs to be instantiated. Then, the mail backend can be driven at a variety of levels. For example, through nsIMsgFolder interfaces (e.g., nsIMsgFolder::GetNewMessages retrieves new messages for a POP3 or IMAP inbox), by running URL's (e.g, nsIMessenger::OpenURL(uri) loads a message into a docshell), or by directly access a message service, e.g., nsImapService::StreamMessage(const char *aMessageURI, nsISupports *aConsumer...). As you'd expect, the higher level functions handle pretty much everything for you, and the lower level functions rely on you to consume the data and do with it what you will.

The more of the existing backend you use, the less code you have to write. If you use the lowest level, you'll get functionality similar to what c-client or Twister give you. But if you use the higher levels, you get a lot more functionality, including account management (servers, lists of folders on server, etc), summary files for individual folders, offline support, etc. All of this is independent of the UI, but does require that you use the Mozilla database (Mork, or at least, the MDB API), the Mozilla mailbox format (Berkeley format), etc.

Non-mail components used by Mozilla Mail:

JS (for preferences), Password manager (for managing stored passwords), Necko for networking, NSPR (though Necko uses it much more extensively than mail does). Mail uses gecko for rendering mail messages and composer for composing mail messages, but that's not required. There are a few UI things that would need to be plugged in like prompts, alerts, status messages but those are behind interfaces that would be easy enough to implement.

I think the decision boils down to what level of functionality you want from your external library. If you want low-level functionality, go with Twister or if Twister has some unforseen flaws, c-client. Then, you'd build up functionality on top of that. If you want a lot of higher level functionality and are willing to use our database api, mailbox format, etc., then you should explore using Mozilla more.

A related issue is what kind of e-mail functionality do you want? Do you want a decent mail client with all sorts of cool Chandler functionality on top of it, or do you want a full-featured e-mail client which handles a lot of subtle things? I'm not prejudging the answer - you'll be able to get a decent e-mail client quite a bit faster, and the Chandler features should be compelling enough that people won't care about the other things. I'll give a couple examples of subtleties:

1. What happens when a user clicks on a large imap message and while it's downloading, decides they want to read another message instead? In c-client and Twister, you need to wait for the whole message to download before you can download the second message. For large messages and/or slow connections, this can take some time. If you were to just drop the connection, then you need to set up a new connection and set up your connection state again, which can also be slow. What Mozilla does is measure the message download speed, and fetch the message in reasonably sized chunks for the download speed, and if the user tries to download a second message, we wait for the current chunk to finish downloading, and then download the second message.

2. Connection management - you have to be careful not to have two simultaneous connections to the same imap folder or the UW server will kill the first connection and lose all the state changes made with that connection. Mozilla goes to a lot of trouble to ensure this doesn't happen. I believe c-client also handles this internally. I don't know if Twister does or not. One simple way of handling this is to only ever have one connection to the imap server - Mozilla caches a user-configurable number of connections so that switching back and forth between folders can be faster.

POP3 Issues

POP3 is simple, but there might be a few complications, besides the aforementioned poorly behaved servers. I don't know if the client libraries out there handle POP3 leave on server well, because that requires the equivalent of Mozilla's popstate.dat, to keep track of which messages have been downloaded, and to handle deleting messages from the server that have been deleted locally. Mozilla's pop code also supports partial download of large messages, which is nice for dial-up connections.

-- MichaelToy - 18 Aug 2003