Chandler Vision

This is a draft, a possible replacement for ProductRoadmapNov2004.

Chandler is an open source Personal Information Management (PIM) client application with innovative design and ambitious plans for sharing, extensibility and cross-platform support. This vision overview or roadmap (Jim McCarthy would call it a "technology plan" or "a Tomorrowland that has credibility") explains our high-level ideas for achieving these ambitions.

Contents

Target Users

Chandler targets info-centric users. Info-centric users receive and send lots of email on a variety of topics and often collaborate heavily with other users. Most of their inputs are electronic text: email, documents, Web pages. Most of their outputs are electronic text: schedules or specifications for development teams, vision statements, purchasing proposals or advertisement campaign ideas. To be effective, info-centric users must be able to manage and organize their incoming, outgoing and stored information.

Info-centric users who are also early adopters of new technology are likely to be the first users of Chandler. These users may introduce Chandler into their workgroups and spread adoption. Since many other people in the same workgroups are not early adopters (or not info-centric workers) we realize that Chandler will have to integrate and communicate reasonably well with existing mail and calendar software.

• Information overload can sidetrack anybody. Both your incoming messages and your information archives can contribute to loss of focus or getting completely distracted.
• Poor interoperability between calendar tools.
• Platform-specific software often leaves you with Hobson's choice of which calendar application to use.
• Information is trapped in one application -- email, calendar and task data live in their separate data stores and can't easily be blended.
• Organizing Email requires too much work. Email applications require all emails to be organized before that work bears sufficient fruit. Email users have to do meta-organization to decide how to reorganize into hierarchies because hierarchies are so inflexible.
• Tracking progress is too hard. Tasks are either done or not done; email is either read or not read. There's no room for making small steps, for having partially completed something.

This document does not promise feature delivery dates or release dates, however we try to indicate which features are in release planning and which are on the radar for later.

Design of PIM Objects and Views

The key features for a Personal Information Management (PIM) tool are email, calendar, tasks and contact management. Chandler adds general Resources to that list, with the idea that email is so integrated with managing and handling information resources that these need to be better integrated.

Email features

For users who make heavy use of email, dealing with email quickly is of primary importance. Chandler tackles that with some non-standard features like Triage for quick handling of the bulk of less-important email (Triage is discussed later in some detail). We also consider worflow-based designs for even the basic features of email in planning:

• View new mail (classic table view) and read individual mail in pane or new window
• Compose mail, reply/forward or new mail, including maintaining drafts
• Synchronize mail for view/use offline (IMAP accounts only of course)
• Read, save and create attachments
• Read S/MIME (encrypted or signed) mail
• Detect and manage spam
• Import/export and printing

Calendar features

A basic calendar application requires the following features:

• Views of events by day, week and month, and navigation by time period
• Detailed view of event, ability to add and remove events
• Recurring events

Info-centric users have a few additional needs just for managing their own events

• Multiple calendars (e.g. a "soccer games" calendar separate from "work") and calendar overlays
• Selection of timezone by view for use while travelling or scheduling meetings that span timezones

Several other important calendar use cases are discussed in the collaboration section of this document.

Notes and Tasks

It's important not to interrupt flow. When a user wants to write down some thoughts or meeting notes quickly, it should be possible to do this without first considering which application and then considering what file name and then what title, etc. Chandler will provide a simple gesture for creating a "New Item" to facilitate flow and recording information. A new item begins life as a note and the only work after creating this note is to type the thoughts that triggered its creation.

Notes often flow seamlessly into tasks. Although in the first few moments of recording some information the user isn't aware of a future task, this is often the next step to follow. Chandler's gesture for this is to add Task nature to the note with stamping (more on this later).

Many info-centric users don't even use task-list applications. Some of the problems:

• Not integrated with email which is where most tasks end up sooner or later
• The easy-to-use systems don't scale smoothly and intimidate users from adding their full list of tasks
• The scalable systems often require learning investment or other high barriers to entry.
• Require too much decision making to create each task. (What folder to create in? What title to give it? When is it... ahh, forget it.)

Task management is integral to Chandler's design, permeating email and calendar workflows seamlessly and scaling through the same mechanisms that are used to manage email.

Contacts management

Contact information is certainly an important part of the user's personal information. Some contact information is required to use email and calendaring functionality:

• Auto-completion for address fields when composing mail
• Auto-completion for calendar addresses for scheduling or browsing calendars
• Auto-completion for sharing calendars or other information or browsing shares

To make auto-completion work we need to provide ways for users to create at least minimal contact records and add addresses, remove addresses or remove contacts.

Chandler will also help users manage, view or find email according to who sent or received the email, as well as events by who organized or is invited to the event.

Contact information like phone numbers, addresses, birthdays and other personal information is important too, but managing that kind of information is targetted after Chandler 1.0.

Resources

A challenge for info-centric users managing rich documents, spreadsheets, or other resources is to correlate information in the file system (which can be pitifully inadequate) with information from email history. The user might be able to go to the file system to see when a file changed last, but have to do separate searches in email to find out when the file was last sent out for review. Chandler will enable users to bring resources into the Chandler repository in order to manage these resources together with the collaboration or communication workflows.

Information Processing

Although the information processing features are related to email, calendaring and tasks, much of information processing and management crosses application boundaries. Chandler tries to help users focus on the content and meaning of information, not on format or what application the information was created in.

Stamping and Heterogeneity

Who says that the lines between emails, events and tasks are clear? Users need to manage their information according to project, not according to application. When you pull all related things together, time spent switching context and hunting is much reduced. Chandler offers heterogeneous collections, able to contain any kind of Chandler item as well as resources that might otherwise live in random places in the file system. Naturally, searches can also cover all application areas at once or alternately be limited to specific kinds of items.

More subtly, we believe it's powerful to allow users to not only put their peanut butter and chocolate in the same cupboard, but also to mix their peanut butter and chocolate together in the same item. We call this stamping, as in "I want to stamp this note as a message" or "I want to stamp this message ask a task". The user adds email-ness or task-ness to the pre-existing item without creating a separate item. Consider some of the possible use cases for this:

• An incoming email leads to an ill-defined task. Rather than have to create a task and try to decide exactly what it is and what to call it, just stamp the email as a task to be sure to come back to it later.
• A co-worker has to be notified about an upcoming event. Rather than create a mail and give it a subject then copy information from the event to the email, just stamp the event as a message and fill in the "To" field.

Stamping replaces the flagging feature that traditional email clients often support. Flagging is tantalizingly close to being useful for many people but lacks the ability to define due dates or detail to explain why something was flagged. Since stamping an email as a task is just one click, it's as easy as flagging and doesn't force a series of decisons. All the user has to know is that there is probably something to do, sometime, and stamp as task. Later the user can remove the task stamp, assign a date, or add more details to the description of what has to be done.

TODO Find an image of a stamped item, an EmailTask^?

Quick Triage

Sorting through new and old email consumes a large chunk of users' time. In the traditional email triage model, the user must perform a certain workflow with every email, even those requiring no action:

1. Select and read the email to decide that there is no action to be taken
2. Decide where to file the email -- typically, which IMAP folder best fits the topic/status
3. Drag-and-drop (mouse intensive!) or other gesture, possibly including scrolling folder list and/or opening collapsed hierarchies

Some users simply refuse to file emails, leaving tens of thousands of emails sitting in an enormous inbox. By taking this shortcut, however, these users fall prey to having to review emails over and over again to determine whether there's something to be done. This is contrary to the advice given by many books and training organizations -- to deal with each received email a small number of times and keep backlog low. Ideally the user would review new email only once and decide immediately whether to delete, file, deal with now or plan to deal with later. However, users consistently fail to deal with email in this manner, even after taking productivity training, and part of the blame can be laid on email tools.

With modern search functionality, it shouldn't be necessary to file emails meticulously into hierarchical sets of folders. Email clients with modern search capability show that it's actually fairly easy to find uncategorized and unfiled emails later.

Chandler will offer very quick gestures for the user to say "Ok, I've seen this now and any action necessary has been taken". An email marked as "done" will not be in the user's face in future triaging sessions and will still be findable as we'll see.

Greater Productivity through Procrastination

So much of incoming information can't be dealt with now. Sometimes we just can't take action yet. Sometimes we need a different environment to read carefully. Sometimes we could take action but shouldn't due to higher-priority work. Many items can and should be dealt with later. It should be easy for the user to defer action on email and have the client ensure that it doesn't get lost. In Chandler the user gives the item a tickler, a trigger that will return the item to their attention later. When an item has a tickler it has been stamped as a task -- this shows how stamping permeates the design. The word tickler comes from the David Allen task management system which suggests a manual technique for maintaining a set of reminders and a habit of looking at them to see which are due.

Putting the deferred status together with the done status, and with the default status of "now", we have a powerful triage workflow for sorting through new email. In addition Chandler will allow users to triage notes, tasks and events in the same workflow and even in the same view. Some examples of use cases justifying this generalization of functionality:

• Having created a note very quickly while the idea is fresh, the note begins with a "now" status (neither deferred nor done). Then while triaging, the note can be left as is, deferred or archived (done).
• A task reminder can pop up for a task that the user now realizes can or should be deferred again to a yet later date -- or if the task has been dealt with the user can archive the task.
• A recent event might remain in the user's dashboard until the meeting notes have been published -- then the user triages the event as done/archived.

Chandler brings the ability to mark as done and the ability to defer together into the Dashboard view. This is the crucial view of what's up now in the user's information space. In this view in particular, users will be able to quickly sort and focus on what's important or not fully considered.

Finding information

One way to find information is to do brute searches over raw data. Another approach is to organize information ahead of time, by categorizing, classifying or tagging information. Chandler will support both for all kinds of items. Brute searches are conceptually simple, so there's just not much to say about our plans for search functionality in Chandler.

Organizing information is a much more interesting topic. There has been much work in this area, from library classification systems to folksonomies created through uncontrolled tagging. Our research and organization design work persuades us that each approach alone has its good points and bad, so we are planning a combination of categorization, unstructured tagging, and structured facets (such as kind, triage status, sender/recipient and date created/received). This combination will be a big win for users trying to locate the information they want at a given moment. How do these all work together in practice?

Users may (but aren't forced to) categorize items. This is done through the classical approach of dragging the item into a collection. Collections are most likely topical or based on content. Collections should not be based on status, sender or time period, because Chandler has affordances to filter those facets independently. Also in Chandler collections aren't hierarchical; all collections exist at the same level. With alternative ways of handling or reducing complexity, users shouldn't need so many collections that a hierarchy becomes necessary.

A couple examples of how facets reduce typical collection explosion:

• Some email folders are based on status instead of (or as well as) topic. A user might normally create folders for "Project A Todo" items and "Project A Done". But Chandler treats status as a facet of every item, so a single topical collection for "Project A" can be viewed according to now/done/deferred status.
• Many email folders are created to hold mailing list traffic. If it were easy enough to filter any set of messages (possibly the entire repository) based on which list the message came in on, the user wouldn't need to set up that categorization manually or even create explicit rules.

In 0.6 Chandler already has a classification-capable Sidebar with the ability to quickly apply Kind filters to any collection or to the entire repository.

Clusters of Items

Sometimes users need a level of organization that's a little more granular than a collection, and a little more unobtrusive. We see this in threaded email readers: a conversation may appear as a single table row that can be opened to view the individual messages. Effectively, a thread is an application-specific cluster of items. We can apply the cluster concept to other items as well.

Clustering tasks together might be a power feature for task-intensive workers. A project might start out as a single task, but as work progresses other tasks are added and possibly scheduled. There's no requirement to start off defining a project when it can grow organically by tacking on tasks one by one. Important resources, messages or events might be added to the cluster if that's the way the user thinks to organize these things. Alternatively, a cluster might not be related by project but by venue or time period: a cluster of things to do while visiting Paris, for example.

Collaboration and Communication

In addition to sharing information simply by sending it (email), we have ambitions for Chandler to make it easier to share and collaborate in many other ways. At the same time, Chandler must remain completely usable if the user takes the application offline at a moment's notice. Users no longer accept (nor should they have to) manual synchronization choices and degraded offline behavior when it's possible for offline behavior to work well.

In contrast to our intention to do the best interaction design we can, we intend to do an adequate job at collaboration. We are willing to make compromises in this area particularly for the 1.0 release. However we believe that many existing calendar and email products are actually not good enough (for example iCal is not good enough at sharing calendars collaboratively). Our vision here rises above the standard level of ambition even though we don't consider it our most important or ambitious goal.

Messaging standards

Chandler supports basic IMAP, POP3 and SMTP and will be able to import locally stored mail. We hope to make these perform well and operate smoothly without much configuration or manual management required. Many info-centric users need a great offline/online experience with no barriers to sudden disconnected operation and automatic reconnection.

A challenge in supporting IMAP and importing mail arises from our collection model. Since IMAP is so focused on hierarchical folder operations we'll encounter some peculiarities downloading importing messages into the Chandler repository. IMAP doesn't support as many facets as Chandler uses (e.g. the now/done/deferred status), so we may have to use IMAP creatively to persist this information in the IMAP server. If other email clients browse a IMAP account with mail managed by Chandler, we can't ensure that the other email client can present the information in a consistent way.

Some of the emerging IMAP extensions and messaging standards offer benefits to Chandler users. Annotations offer the ability to synchronize custom information. The "sieve" system for running mail sorting rules on the server can make synchronization faster. We hope that mail servers decide to add support for these standards soon.

Instant messaging is not part of our immediate messaging solution though we certainly think it would integrate wonderfully into Chandler, even as an independently developed parcel.

Server-hosted sharing

Info-centric users need to share various kinds of items: resources like spreadsheets and images, events or whole calendars, task lists, and sometimes even email. Sometimes these are best shared mixed together: contacts, messages and events relating to a collaborative project, shared read/write amongst the collaborators. A share should be flexible enough to be public, private or offer limited access; information should always be available, and it should support multi-author operations smoothly handling synchronization and conflicts.

The emerging standard for sharing calendars is CalDAV. CalDAV is based on the general Web-authoring and file sharing protocol WebDAV which can be used for sharing heterogeneous collections of items. Chandler sharing via WebDAV was experimentally functional in 0.5. In 0.6 we partially support CalDAV and have improved the sharing GUI.

The workflows for sharing are currently focused on sharing collections, although we envision sharing individual items as well. The sharing workflow starts from choosing a collection to share then picking who to share it with. Chandler automatically sends an invitation to each read or read/write participant, then keeps track of each participant. On the receiving end, Chandler can automatically handle an accepted invitation, converting it into an automatically synchronized collection.

On the radar, we have ideas for browsing shares on servers, more secure sharing, including better ways of managing access control, possibly public key exchange and/or encrypted shares.

Scheduling

There's more to collaborating around calendars than simply sharing the calendar. Schedulings features have become vital for people whose schedules are moderately to very full -- and for the people who have to work with busy collaborators and somehow fit into their schedules.

Required features for calendaring and scheduling:

• Share just free-busy time to collaborators who don't require access to the full calendar
• Read/write access to shared calendars for closer collaborators
• Send and receive invitations; accept and decline incoming invitations; review attendee status

OSAF is looking into iMIP (iCalendar invitations in email) as well as proposals for CalDAV server-mediated scheduling.

On the radar:

• Scheduling delegation/designation
• Room and resource booking
• Assigning tasks to other people
• Views specialized for multiple-person task systems (project management)
• Innovative scheduling workflows (instant messaging?)

Identity

Identity, authorization and authentication has turned out to be one of the most challenging parts of the sharing work described above. Traditional solutions suck -- it's hard for the user to manage their own accounts in various locations, let alone manage the email, IM, blog and sharing addresses of all their contacts. Much work on identity is going on now, and has even been called a big-bang. We are tracking this work.

The standard solution for server sharing is to use access control lists to grant permissions to named users. However this has limitations:

• The GUI experience for this is often terrible. At a minimum, a usable sharing workflow must manage permission changes for the user together with notifying/inviting the sharing participants via email, because managing permissions and notifications manually and/or separately is so painful.
• It's difficult to correlate a collaborator's messaging address together with their sharing identification. Monolithic collaboration systems like Yahoo Groups simplify collaboration by centralizing all functions so that the server can manage all accounts and addresses. Achieving the same usability in a de-centralized system is a worthy challenge.
• In a de-centralized system there's no guarantee that a collaborator even has an account on the server where the sharer has an account and has created the share.

As a low-cost (albeit not very secure) usable solution to some of these problems, Chandler uses anonymous tickets to grant access to shares. The sharer defines a list of viewers or collaborators and Chandler sends share invitations to these addresses together with appropriate access tickets.

On the radar

• Support for Shibboleth or some single-sign-on solution for universities.
• Further ACL work related to WebDAV and CalDAV servers
• Support for creating groups of principals (for sharing or invites)
• Contacts management linked with sharing management
• Automated ways to discover sharing identities from known email addresses or vice versa
• Ways to manage accounts more usably (auto-configure?)
• Browsing/searching for user accounts on sharing servers

Cross-platform

OSAF's vision is to make Chandler a native cross-platform application, one which looks terrific on all supported platforms without the need to port the code. We intend to improve the state of the art in this area, creating the tools/widgets/libraries ourselves when necessary and contributing improvements back to the Open Source community.

Advanced and Specialized Widgets

Although we are using a common widget set (wxWidgets, which has a Python wrapper called wxPython), we find that our ambitious usability goals and user interaction designs require custom widget development Note that even if we were using another widget set (in 2003 and again in 2005 we investigated others, including XUL via pyXPCom) we would still have to create custom widgets and contribute improvements to raise the level of visual polish across all platforms.

• A mini-calendar widget that supports selecting date ranges, and that shows a visualization of "business" level for each day
• Date and time entry fields that support typing and can remain in "undecided" state
• Combination checkbox and icon widget for the sidebar so users can select multiple collections for simultaneous view (e.g. several calendars overlayed)

High-level GUI control

Because wxWidgets is a moderately-level display and interface framework, we've designed a high-level framework to use it -- the Chandler Presentation and Interaction Architecture (CPIA). The "customer base" of the high-level framework includes both parcel developers and those who develop core Chandler GUI. In the long run, a well-designed CPIA will speed development of new GUI.

Extensibility

What makes open source projects not just good, but great? One factor we've seen is a good capacity to accept contributions, not just to the main code-base but also through extensions. QuickSilver^? has quickly become a necessity to many of its Mac adopters because each user can download the extensions that match the way they work best. Eclipse will maintain high relevance as an IDE because its plug-in architecture allows it to be extended with new features not only for Java development, but also for other languages and syntaxes (e.g. Python code and XML files). We believe this extension model is particularly appropriate for a PIM because although there is a core of functionality found important by many users, there is also a long list of features that only a few users find to be critical. Features that are critical to a few and unused by most are ideal for development as parcels for Chandler.

Email, task lists and calendars in particular are areas where users demand specific functionality. Many people have become stuck using a particular email client because it provides one critical feature. Even if there are several functional limitations of that client, that one critical feature creates lock-in. Upgrading might become possible if only there were a way to recreate that feature in the new email client.

Todo: review with Mimi because she said some stuff about extensibility that I'm not capturing.

Parcels

Our vision for Parcels is to have an easy mechanism to contribute new areas of functionality to Chandler. We plan to support third-party parcel development and distribution both independently and through centralized services. The user will be able to easily install or remove parcels or just review what parcels are installed.

Chandler 0.4 supported parcels, including an example RSS newsreader (ZaoBao^?). Chandler 0.5 improved support and test parcels could share photos on Flickr or view bugs in Bugzilla. In 0.6 we eliminated XML for parcel schema definition as we continue to strive for ease-of-development, because we found that learning an XML data definition format in addition to learning Python was mostly an added barrier to parcel developers and not enough of a benefit. Instead we developed an Python API for quickly defining parcel data types and GUI layout, and naturally continue to offer Python APIs for repository use, event handling and GUI changes.

Data models, content models

Careful modeling work makes complex systems understandable. In Chandler we pay attention to the data model (the raw repository, or how to store arbitrary items and how to find, see, move, delete, change or extend them) and to the content model (what is an email, what is a task, what are the attributes of a shared thing). Modeling is a difficult task -- the needs of the moment encourage special-casing, inconsistency and complexity. Yet the long-term needs of enabling third-party extensions (and making them understandable and coherent in our user interface models) requires that we keep our models simple, clean, consistent and rigorous. In fact even without parcel APIs, it's a long term boon to maintenance and internal extensibility to do rigorous modelling.

In the point releases leading up to 0.4, an extensive content model was created before we had code or clear specifications to back it up. In 0.5 we rethought attribute indirection and stamping models, and in 0.6 we have reviewed and reworked the repository API. We continue ask ourselves

• How have our models evolved?
• Did we overlook any major simplifications?
• Have we degraded consistency and do we need to clean up?
• etc.

Repository and Extensibility

Open Source development, together with our parcel plans, present a few challenges specifically in architecture of a PIM repository. We considered using a traditional or object database directly, without abstracting that at a higher level. However, we foresaw that many features that ought to permeate the user's information (collections, clusters, other relationships between items, facets and tags) would be difficult to maintain for and with extensions, unless we built a specialized higher-level repository API. Thus:

• Native and custom items can all belong to any collection or cluster.
• Any kind of item can link to any other kind of item -- a custom "Support ticket" item could reference a native email item and vice versa.
• Custom items can be tagged and support facets just like native items.
• We can display custom items in summary views and repository browsers without making the parcel developer do extra work.

This is not an exhaustive list but provides an idea of the costs we'd impose on parcel developers to integrate well if we used a naive storage approach.

The same ideal applies not only to the repository but also to our collaboration routines, protocols and data formats. Custom items can be shared and access control applied using existing mechanisms, without extra work from the parcel developer.

Enabling hacks and exploration

Our ideal for Chandler extensibility is to make it incremental, allowing small and quick wins to bring an idea gradually to the stage of becoming a great individual feature or widely distributed parcel. We'd like to allow even pseudo-programmers (those who don't want to embark on software architecture but know how to string a loop together with a conditional statement) to automate their own personal processes with a bit of easy scripting. Even non-programmers can benefit from these mini-extensions if the programmer shares their few lines of code, provided there's an easy avenue for tying this script into Chandler.

The choice of Python for Chandler development was in part due to this ideal. Python is a scripting language but also scales up to be a full-fledged programming language. With Python, we won't have a discontinuity or learning gap between scripting and parcel development or even for contributing to the core code-base.

Other features enable code/API exploration along with scripting. We already have a repository viewer tool which allows a scripter to explore and view the results of script execution and easily iterate on their work. Often a tool will have immediate use within Chandler development, and if we polish it a little it will be useful externally as well.

Community development work

We aim to foster communities of users, scripters, parcel developers and contributors (whether contributing QA, documentation or code in large or small quantities). Some of the pieces needed for that:

• Open Wiki for project information, documentation and any product thoughts
• Public mailing lists
• Open IRC channels for developers to meet and chat
• Documented APIs
• Tutorials, both online and taught at conferences
• Programming sprints at conferences
• Bug bashes and bug validation sessions on IRC
• Contribution review processes, open bug database