This page is a draft, a possible replacement for ProductRoadmapNov2004.

Executive Summary

Chandler is a open source Personal Information Management (PIM) client application with innovative design and ambitious plans for sharing, extensibility and cross-platform support. This roadmap (Jim McCarthy would call it a "technology plan" or "a Tomorrowland that has credibility") explains our high-level plans for achieving these ambitions, including the challenges we're already facing and expect to face. We sequence these plans into major releases which may each take couple years. The three major Chandler releases we discuss are called Kibble, Westwood and Pasadena.

Kibble is named after the "dogfood" concept that you don't really know what it is you're making until it's used, and the first users should be those who make the product (whether that really should extend to makers of dogfood is arguable but we believe in this principle for software). Kibble is being designed for enthusiastic, small-organization or individual early adopters -- basically, us.

Westwood is targetted to use in universities and similar institutions, with more advanced organization needs and less enthusiasm for software that has promise but also has problems. Westwood is intended to help us cross the chasm by aiming for an audience that is somewhat of a niche but definitely on the way to broad adoption.

The current idea for Pasadena is to aim for broad adoption.

Contents

Innovative design for information management

Chandler was founded on an innovative design vision. This is the most important thing to understand about the Chandler development plans because implementing this design is a significant part of our early work.

• Most of the innovative design (so far) is being implemented in Kibble. Westwood will see an iteration on that and we can't yet predict what other design work will be done in Westwood or in Pasadena.
• We haven't seriously tackled the design challenges of making Westwood and Pasadena features usable (for example, finding people in your organization, on your server, or elsewhere, to share with -- or more fundamentally, making identity usable).
• Innovative design requires serious GUI widget work which is extending our schedule well beyond what was originally imagined. We address this in part by hiring more GUI developers but there is a limit to the practicality of simply adding developers.

Note that we are targetting info-centric users with this interaction design. We're not trying to optimize information management for somebody who receives ten emails a month or even ten a day. Info-centric users receive and send lots of email on a variety of topics and often collaborate heavily with other users. Their need to find better ways to manage their information is what will drive them to try Chandler, and we must put in place the features that will make them keep Chandler once they've tried it.

Finding Information

There are two approaches to finding information. One is to do brute searches for it without preparation. The other approach is to organize information ahead of time, by categorizing, classifying or tagging information. Brute searches are conceptually simple, so there's just not much to say about our plans for search functionality in Chandler -- it's just work. Much of the initial work is in Kibble and we don't yet know what refinements we might consider in Westwood and Pasadena.

Organizing information is a much more interesting topic. There has been much work in this area, from library classification systems to folksonomoies created through uncontrolled tagging. Our research and http://wiki.osafoundation.org/bin/view/Projects/OrganizeDesign^? persuades us that a combination of classification (without much hierarchical navigation) and unstructured labelling, with some important pre-determined structured facets (such as kind, triage status and when), will be a big win for users trying to locate the information they want at a given moment.

The organizational design of Chandler affects the development of the sidebar (navigation functionality), filters, searches and views.

In 0.6 Chandler already has a classification-capable Sidebar with the ability to apply Kind filters. We have a significant amount of work remaining to do in Kibble to enable quick filtering and selection of the remaining built-in facets. One challenge presented by this innovation is that we can rarely make use of existing GUI controls -- the traditional hierarchical tree GUI does not at all meet our needs, and thus we must develop rich widgets specially for Chandler.

In Kibble we also need to extend our search capability so that users can find information without organizing it. We plan to support full-text search on any collection/view and on the entire repository.

Triage, Dashboard and Ticklers

One of the most common tips for dealing with information overload is to attempt immediate triage. Books like Getting Things Done^? and training organizations like McGhee recommend to email users to deal with each email a small number of times and keep backlog low -- ideally decide immediately whether to delete, file, deal with now or plan to deal with later. However, users consistently fail to deal with email in this recommended manner and part of the blame can be laid on the tools they use.

• It isn't easy enough to decide and remember to deal with email later. Users worry that they'll forget to deal with an important issue in time so they leave it in their inbox. That means they spend time scrolling through old email looking for cues that there's something they must deal with immediately -- yet there are very few tools to avoid that scrolling or to make the cues obvious.

• It isn't easy enough to decide that one is done with an email and stop having to worry about it. Instead of simply allowing the choice "Now that I've seen this email I never have to see it again" most email clients force users to categorize the email in order to file it in an appropriate folder. Categorizing an email can be difficult enough that some users never use categorization.

To deal with these two problems we plan to implement management dashboard for triaging all types of items in one place. Part of the goal of the dashboard is to see "what I need to pay attention to" in a single place, combining tasks and emails with upcoming events. But the other major goal of the dashboard is to make triage trivial. Users should be able to archive mail by making only the single decision "I am done with this" and Chandler removes it from the dashboard forever (or unless they change their mind). Users should also be able to quickly decide "I need to deal with this in a week|month|other" and Chandler immediately removes it from the dashboard until it needs to return. The word "tickler" comes from the David Allen system. Allen outlines a plan for manually maintaining a set of reminders and a habit of looking at them to see which are due -- an obvious task for PIM software assistance.

The dashboard is not yet started as of point release 0.6. However, in Kibble we want the dashboard to be at least usable within OSAF. That means that in the next two point releases we must focus on dashboard and iterate on its design.

In Westwood we will certainly iterate on the dashboard and Pasadena work in this area is unforeseeable.

Crossing Item Type Boundaries

Existing PIM clients tend to isolate information according to type. That is done for the convenience of the developer and designer, not for the convenience of the user. There are several ways in which Chandler intends to allow users to cross those traditional boundaries between "email application" and "calendar application".

Stamping is the item-level way to do both email and calendaring. In 0.6 a note can be stamped as an email, then as a calendar item. An event created through the calendar view can then be stamped as an email to send it to somebody.

User-designed collections means that if a user wishes to organize email and calendar information together Chandler does not put barriers in the way.

These innovations are already resulting in a non-insignificant amount of extra work.

• Already we have reworked the stamping implementation at least once because of dissatisfaction with the first programming model. Making stamping work together with extensibility requires getting this right.
• We need to put extra thought and work into combined views. It's more difficult to appropriately display email, calendar items and tasks together than it is to display them separately. We are concerned with showing the appropriate layout (tables?) and the appropriate information in each column for emails, events, tasks, and for stamped (multiple-type) items.
• We need to make sure filtering by kind works well -- quickly and easily. Because we allow a mix of item types, yet we know that sometimes the user will be searching for an event or an email, we need to allow the user to quickly eliminate from view the types of things they are not looking for.

In 0.6 stamping and heterogeneous views are at the plausible-promise stage. In Kibble we need to get them to the final (usable) stage. There is also still a serious risk of redesign or rework for this innovation, in Westwood or possibly even in Kibble.

Active Help

Although we're fans of help, tooltips, context sensitive help and user agents, and had initial ideas in this area, those ideas are deferred in order to make progress first on the core four visions.

Kibble: Basic help files.

Westwood: Polished help.

Pasadena: Context sensitive help, user agents.

Collaboration

In addition to sharing information simply by sending it (email), we have ambitions for Chandler to make it easier to share and collaborate in many other ways. Despite the heavy use of the network however, Chandler must remain completely usable offline at a moment's notice. Users no longer accept (nor should they have to) manual synchronization choices and degraded offline behavior when it's possible for offline behavior to work well.

In contrast to our best-possible approach to user interactions, our ambitions to support collaboration are to do collaboration well enough. We are willing to make compromises in this area. Note however that we believe that many existing calendar and email products are actually not good enough (for example iCal is not good enough at sharing calendars collaboratively) so our level of ambition here still rises slightly above the average.

Messaging standards

Chandler supports basic IMAP and POP3 now and we have more work to do to finish this support and make it broadly interoperable in Kibble. To a certain extent we rely on pre-existing protocol libraries for this work but we've found those need some testing and investment on our part.

A challenge in supporting messaging standards is that these protocols were designed to support very specific usage models. IMAP was designed for

• Hierarchical collections and classification of archive email into that hierarchy
• Searching within a single collection (knowing where in your hierarchy to find something)
• Deleting items in place and using undelete to change your mind or expunge to confirm
• Adding any custom information to an email such as a color display attribute or "due date" stamp

It takes extra work to support IMAP when diverging from these models -- when we don't encourage hierarchical collections classification management, in particular. We also need to implement search and indexing locally (it's not clear at all if we'll use server search functionality) because for Chandler, search needs to work across kind categories and across IMAP folders and while offline as well as online.

As of yet we don't know how we're going to handle emails with extra information such as what color to display the email in, or for that matter, emails stamped as something else. It may be error prone to try to synchronize those emails to the IMAP server and keep the extra information locally only, and besides that doesn't allow the user access to the extra information through a WebUI^? or from another client. Yet synchronizing that information to the IMAP server is very hard.

Some of the emerging standards are very interesting -- annotations, in particular, offer the ability to synchronize extra custom information, provided we see servers supporting annotations (as we near last call on this proposal there is only one server implementor promising to implement annotations at all). Even if server implementations come quickly, deployment will still take considerable time so we will still need to interoperate with servers not supporting annotations.

The difficulty in supporting email well together with innovative design means that we do not necessarily expect our email functionality to be broadly usable when we reach Kibble -- it may only be usable at OSAF or even show plausible promise but not yet be able to replace mature email clients. Some of the factors:

• Power email users may have more need to classify than we're aware (even when presented with other tools)
• Spam filtering, spam flagging
• Inbox rules -- editing and applying
• Displaying and editing formatted email (HTML)

Serious work on email, messaging standards and messaging interoperability will continue into Westwood.

Server-hosted sharing

The emerging standard for sharing calendars is CalDAV, based on the general Web-authoring and file sharing protocol WebDAV. Chandler support for WebDAV began in 0.4, with sharing experimentally functional in 0.5. In 0.6 we are adding the first steps of CalDAV support and improving the sharing GUI.

In Kibble we must

• iterate the sharing GUI experience until it's usable
• Finish CalDAV support
• Decide which WebDAV servers to support and test and fix interoperability
• Decide whether to do ACL or tickets for sharing permissions
• Decide what to do with servers that don't support ACL or tickets)

This looks like a full-time resource assigned to sharing for the rest of Kibble. Note that Kibble sharing allows people to share their calendars, share group calendars and (if they have permissions) add events to shared calendars.

In Westwood and Pasadena we are not sure yet how we will extend sharing though we have ideas for

• Very secure sharing, based on public key exchange and encrypted shares

Calendar (and task) Collaboration

Some calendar-related collaboration functionality is extremely specific to calendar uses cases. Most of this won't be seen until Westwood and Pasadena. The exceptions are the ability to view somebody else's free-busy time, and the ability to send and receive iMIP invitations.

Westwood: advanced group calendaring

• Delegation/ designation
• Scheduling via server
• Room and resource booking

Pasadena or beyond?

• Assigning tasks to other people
• Views specialized for multiple-person task systems (project management)
• Innovative scheduling workflows

Identity

Identity, authorization and authentication has turned out to be one of the most challening parts of this sharing work. Traditional solutions suck. Much work on identity is going on now, and has even been called a big-bang. We are tracking this work.

Kibble will only include very basic authorization work due to the identity problems

• Tickets
• Possible very basic ACL management

Westwood

• Support for Shibboleth or some single-sign-on solution for Universities -- note challenges here due to lack of standardization of HTTP authentication mechanisms beyond Basic and Digest
• Further ACL work related to WebDAV and CalDAV servers
• Figure out how to correlate identities: somebody's email (to send an invitation) with their sharing server sign-on (to grant them permission to view event)
• Support for creating groups of principals (for sharing or invites)
• Browsing users
• Contacts management
• Linking contacts to other concepts of identity

Pasadena or beyond? Unknown

Other collaboration

Instant messaging is not in the plan before Pasadena. Maybe somebody will write an instant messaging parcel.

Cross-platform

The decision to make Chandler a native cross-platform application has been an expensive decision and will continue to be so.

GUI widgets

The combination of choosing to support several operating systems and simultaneously develop innovative user interface constructs involves extensive development costs.

The choices for Python GUIs were not extensive. We examined the possibilities (yes, including XUL via pyXPCom) in 2003 and decided that wxWidgets already basically worked on our target platforms (though Mac support had problems) and had a strong ongoing development effort and support system, and supported Python through the wxPython wrapper. We knew from that point that we would need to contribute to wxWidgets and wxPython:

• to bring the Mac support to the level we required
• to add brand-new widgets
• to make existing widgets work in slightly new ways
• to fix bugs

Since hiring a full-time developer dedicated to this work in 2004, our investment has continued steadily and unsurprisingly we're seeing serious improvement. In addition to the full-time resource we sometimes allocate other developers (including a summer intern) to this work.

High levels of investment in wxWidgets must continue through the end of Kibble and most likely well into Westwood. The alternative would be an extremely costly and disruptive switchover to another system that would itself require investment to support our unique GUI designs.

Higher-level GUI control

Because wxWidgets is a moderately-level display and interface framework, we've had to design a high-level framework to use it -- the Chandler Presentation and Interaction Architecture (CPIA). The "customer base" of the high-level framework includes both parcel developers and those who develop core Chandler GUI. In the long run, a well-designed CPIA will speed development of new GUI. In the short term it's a significant investment to develop the framework itself.

CPIA development work began in 0.1 and continues to be a constant investment of a resource or two for each point release up to and including 0.6. [TBD: evaluate ongoing load].

Libraries

Because everything in Chandler must work across platforms, we are often in the position of being unable to use libraries or capabilities built into an operating system. For example, we can't use Windows system calls to select or list timezones, because Windows does timezones slightly differently than Mac and Linux do. Instead we must find or develop a Python library or a Python wrapper for another library. Then we must integrate that into our build, download and install -- and manage upgrades to that library.

Because of our extensibility ambitions, we must also consider how libraries that we use may be used by parcel developers.

Library usage and integration work in Kibble is a continuing ongoing load spread across OSAF (with significant investment by our build engineer) and will continue through Westwood and Pasadena.

Extensibility

What makes open source projects not just good, but great? One factor we've seen is a capability to accept contributions, not just to the main code-base but also through extensions. QuickSilver^? has quickly become a necessity to many of its users on the Mac because each user can download the extensions that match the way they work best. Eclipse will maintain high relevance as an IDE because not only are its Java plug-ins excellent and extended with new features as Java itself grows, but because outsiders have also created plug-ins to edit and develop Python code and XML files. We believe this model is particularly appropriate for a PIM because although there is a core of functionality found important by many users, there is also a long list of features that only a few users find to be critical. Features that are critical to a few and unused by most are ideal for development as extensions.

Extensibility is a section unto itself in this roadmap because this dream not only involves a significant fraction of effort directly involved in making extensibility possible, it also affects how we do other work.

Parcels

The parcel framework was begun early and somewhat updated in 0.4 and 0.5 releases. At the end of 0.5 it was possible to quickly develop parcels to share photos on Flickr or view bugs in Bugzilla.

In 0.6 we seriously rethought and reworked our parcel definition architecture. We are eliminating XML, reasoning that for a parcel developer, learning an XML data definition format in addition to learning Python simply increased the barriers to creating parcels. Instead we developed an API for quickly defining parcel data types and GUI elements in Python.

Moving towards Kibble, we know we have a serious amount of polishing and documentation workload to ensure that parcel developers can figure out what to do without direct hand-holding.

Looking beyond into Westwood and Pasadena, we have less clear plans. We expect that maintaining a set of public APIs for parcels will require ongoing maintenance work. For example, if we decided to change our stamping model we'd run the risk of having to update the relevant API, which can mean maintaining two APIs temporarily (one of them deprecated) for backwards-compatibility with pre-existing parcels. Being a platform is always costly.

Data models, content models

Careful modeling work makes complex systems understandable. In Chandler we pay attention to the data model (the raw repository, or how to store arbitrary items and how to find, see, move, delete, change or extend them) and to the content model (what is an email, what is a task, what are the attributes of a shared thing). Modeling is a difficult task or at least one that OSAF has found very challenging. The needs of the moment encourage special-casing, inconsistency and complexity. Yet the long-term needs of enabling third-party extensions (and making them understandable and coherent in our user interface models) requires that we keep our models simple, clean, consistent and rigorous. In fact even without parcel APIs, it's a long term boon to maintenance and internal extensibility to do rigorous modelling.

In the point releases leading up to 0.4, an extensive content model was created before we had code or clear specifications to back it up. That has mostly had to be rewritten in 0.4, 0.5 and 0.6. In 0.5 we rethought attribute indirection and stamping models. We have many clues to lead us to believe that this rework is not done -- in fact we should explicitly plan in the remaining Kibble point releases to allocate the time to look at the big picture (more than once) and ask ourselves

• How have our models evolved?
• Did we overlook any major simplifications?
• Have we degraded consistency and do we need to clean up?
• etc.

It's very important that the data and content models be in excellent shape and well-documented when we release Kibble because this is the point where we plan to officially encourage parcel development.

Looking towards Westwood and Pasadena, we expect that new features will often involve new content modelling work. We can't predict the need for rework beyond saying it's a possibility.

Repository and Extensibility

If we were building Chandler within an enclosed team we would have a smaller need for a custom repository. We could use some existing tool like a traditional or object database without abstracting that at a higher level. Instead, the need to support parcel developers requires us to create abstractions. With an abstraction customized to Chandler use cases:

• We can handle permissions internally instead of making them the responsibility of the parcel developer.
• We can share arbitrary items without making that the responsibility of the parcel developer.
• We can display arbitrary items in summary views and repository browsers without making the parcel developer do extra work.

This is not an exhaustive list but provides an idea of the costs we'd impose on parcel developers if we used a naive storage approach.

The repository has been reworked in significant ways early in Chandler development and at the moment has quite stable interfaces but has not yet achieved acceptable performance.

In the remaining Kibble point releases we need to continue working on repository performance and also add to and refine the search/filter capabilities as the features exposed to the user make more demanding use of those capabilities. This is expected to involve the continuing investment of two developers full-time.

In Westwood and Pasadena we do not yet know what additional demands new features will put on the repository.

Communications and Extensibility

The idea that any item in the Chandler repository can be shared, imported/exported and can be emailed imposes a burden on us to integrate communication support deep into Chandler's content model -- yet not so deeply that we can't support multiple protocols easily (e.g. WebDAV vs. CalDAV, POP3 vs. IMAP). In addition, we need to make these features available to parcel developers in such a way that they can extend sharing and email without having to rewrite or replace the functionality we've already created.

We haven't yet figured out how to make this work so we expect it will be a significant amount of work for Kibble and Westwood. Adding new protocols (like LDAP and instant messaging) in the Westwood and Pasadena timeframes will add to the burden.

Enabling hacks and exploration

The choice of Python for Chandler development was in part to enable contributions, both to the core code-base and through optional extensions (parcels). Python is a scripting language, which will allow us to someday enable trivial extensions: a user can create a half-dozen line of code to be run every time an event changes, for example. These kinds of mini-extensions are accessible even to non-programmer users if the programmer shares the few lines of code (and where to put them) publicly.

Other features we develop enable exploration along with scripting. We already have a repository viewer tool which allows a scripter to explore and view the results of a script execution and easily iterate on their work.

CPIA Script is another such tool... [TBD]

Our plans in this area are quite vague and really should be called ideas. However, Chandler developers are well aware of the goals here and well positioned to throw something together that works. Often a tool will have immediate use within Chandler development, and if we polish it a little it will be useful externally as well. We expect a couple projects or other efforts in this area in the Kibble timeframe but haven't had to close down on exactly what this will involve.

Community development work

Building a community, communicating with it and dealing with contributions, will become a serious amount of work:

• Documenting APIs
• Designing and running tutorials and sprints at conferences
• Involvement in public mailing lists
• Reviewing contributions

We certainly hope the community will grow and thus our time spent involved with the community ought to increase. This could easily happen at the end of Kibble (if we are successful in building something people want to start building on, and if we succeed in making Kibble a relatively stable platform).

In Westwood and Pasadena we expect continuing community load but also hope for a return on this investment.

Unplanned and out-of-scope work

A laundry list of features and a rough indication of how unplanned or out of scope they are. These are all naturally candidates for independent parcels or for other open source organizations to tackle.

Even if these ideas are out of scope for OSAF to work on -- and perhaps particularly because we don't intend to work on these ideas -- these may be great ideas for Chandler parcels or for standalone projects that integrate well with OSAF's work.

Feature	Characterization
A version of Chandler for mobile devices	Out of scope
Kiosk mode -- a way to run Chandler on a shared machine without compromising personal information	Out of scope -- see Scooby? for alternate solution to underlying problem
Instant messaging client	Probably out of scope
Advanced event recurrence management (beyond the level of what's in iCal for example)	Out of scope
Signatures and email templates	Not in the plan
Contact management approaching CRM	Out of scope
Peer to peer	Not completely ruled out of scope but doesn't have a clear place in the plan.
CAP support (old calendar standard)	Not in scope
many more things to add...