Dave Crocker on Chandler's anti-siloing goals

I got this email from Dave Crocker on a mailing list discussing messaging futures, when I mentioned how challenging it was for Chandler to do IMAP and also do email integration with other content items. He gave me permission to post his thoughts here on our Wiki.

Many thanks for mentioning this. One of the great bits of technical 'tussle' is the degree to which functions should be tightly integrated, versus permitted to interact through some sort of interface.

My own view is that the tendency to push messaging, calendaring, contacts, etc. all into a single entity is a very large mistake. They need to be able to interact well, but they involve very different areas of expertise. By being tightly integrated, we lose the ability to grow them independently (and to have competition for the components.)

We also tend to obtain a large, cumbersome least-common denominator. At best it serves the most popular needs adequately, but handles minority requirements poorly.

Each of these topics is pretty complex. I think our goal needs to be to find ways to have each performed by a variety of 'products' that vary from very simple to very powerful.

This is particularly important as we need seamless functionality across a wide range of devices and connectivities.

I want a common functional and data core as I move from limited mobile with highly constrained access, delay and bandwidth, to my laptop, with more capabilities, but still-limited communications, to my desktop with high-speed connectivity.

--Dave Crocker

IMAP mental model review

The IMAP model is part of the landscape that an email client must be familiar with and handle. Enough people use multiple IMAP clients that we have to consider the interactions and intentions.

Use cases comments

• IMAP doesn't provide a way of marking messages as Junk. There is no way for one client to mark a message as Junk on the IMAP server and let another client see that flag. Marking messages as Junk is done locally.

• If Chandler marks messages for deletion, and another IMAP client also marks messages for deletion, there's no way to ask the IMAP server which client marks the messages for deletion. So first of all Chandler has to decide whether Chandler is ever going to use this IMAP feature, and then decide when to expunge.

• Since deletion is not a one-step process, the question "I delete emails using another IMAP client... how do other IMAP clients deal with this issue?" is not a sensible question. Instead, we need to be clear how we deal with messages marked for deletion (which still show up and can be downloaded from the server ) and messages which were downloaded previously but are now expunged. Also note that this question appears at least three times in different wordings.

• Cancelling an account and having it deactivated or removed are all really the same case, aren't they? If we're talking about a user action (e.g. the "remove an account" option in Mac OS/X Mail.app), then we can know that the account is gone but it may be only temporary if the user wishes to recreate the account shortly (this happens often enough to deal with administrative changes or glitches). If the account becomes deactivated we'd have no way of knowing without user action, the account would simply be unreachable. I believe IMAP and POP3 have no way of automatically telling the client "that account is deactivated" -- it simply appears to be unreachable and could be due to some other cause. So unless the user tells Chandler to remove the account, we have no choice but to continue to try to contact the account.

• "What happens when you move an email from a folder belonging to one IMAP account to another folder belonging to a different account, possibly residing on a different server?" In this case, the IMAP client actually downloads the email from one server and uploads it to the other. There is no server-to-server communication.

• "How do IMAP filters interact with Chandler rules? Can we translate IMAP filters into Chandler rules?" -- This question is too vague to be answerable. What is meant by "IMAP filters" -- the IMAP SEARCH command which allows the client to get headers for a list of emails which match a search criterion? And what are "Chandler rules", are these rules like Outlook's Rules Wizard? ... IMAP protocol does not have a way of automatically moving messages from one way to another. Spam filtering clients and Rules Wizards (like Outlook's) operate by downloading the email headers, and possibly the body; then send an IMAP request to the server to mark the message (for deletion, most likely) or move it to another IMAP folder -- or the client could simply mark it locally as spam.

• "A Chandler Collection can contain non-email items, should those be reflected on the IMAP account in some way?" -- This question presumes that there is some way to reflect non-email items in an IMAP repository. There isn't really -- everything in an IMAP repository is presumed to be an email. The closest we could get to representing an event is uploading an IMAP object with a iCalendar attachment, and other IMAP clients could interpret this as a draft email and try to send it out or do something else wierd with it. IMAP is really intended as an email repository that the server populates with incoming mail, and the client managers. It's not intended for the client to create new resources, particularly non-email resources. Besides, this kind of feature is definitely not a core or basic feature for an email client, and I'd argue not an important feature dogfood wise.

Proposals

• Do we need to figure out what happens if the user has two instances of Chandler (e.g. one on a laptop, one on a computer) and synchs both to the same IMAP accounts? If one instance of Chandler stamps an email as a TODO, how does the other instance of Chandler see it as a TODO? IMAP doesn't offer a standard way to do this although there may be hacks that could work against some servers.

Suppositions

• the supposition regarding junk mail is factually incorrect as there's no IMAP way to "mark as junk".

• "All incoming mail goes and stays in your Chandler Inbox (unless explicitly removed from the Chandler Inbox collection) even if you've moved it to another IMAP folder from a different IMAP client" --> This plan has serious performance implications -- it means that the client must do many more IMAP operations in order to find out what to display in "incoming". We also need to think through what this means to the user who is trying to clear their plate.

• "Edits to emails in Chandler are not propagated back to the IMAP server." If not, how do two chandler instances sharing the same IMAP account work together? How do we save the user from doing everything twice?

-- LisaDusseault - 07 Sep 2004