Notes from the Chandler Demo

This is a very rough merged transcription of what Ducky Sherwood thinks she heard at the BOF and subsequent talk given at O'Reilly Emergent Technologies Conference, 23 and 24 April 2003, by Mitch Kapor and Andy Hertzfeld. It doesn't have absolutely everything they said because, well, I don't type that fast.

(Mitch Kapor speaking)

Thank you all for coming very much. This is the session on Chandler, the personal information manager. I'm Mitch Kapor, the founder of the Open Source Applications Foundation.

This past week was really exciting for us, since on Monday we released our 0.1 version of Chandler, and we had 13,000 downloads in the first 24 hours. We have a demo that we'll be showing you, but first we want to really talk to you about our plans.

Our mission simple; we're about creating applications in the Open Source model with a real emphasis on quality.

Why are we doing this? Well, I'm a software designer at heart. Email had become my major personal productivity application but and I'd been feeling like there was this gap between what my email client did and what I wanted it to do. This isn't particularly the fault of the email clients, they just weren't designed to handle as much email as they do now.

I realized that email wasn't the only issue. For medium and small organizations, there isn't a simple way to do shared collaboration like calendaring. My wife has a small business, and in order for her to do shared calendaring, she'd have to get this big complex piece of software.

I saw that marketplace didn't seem to be addressing these issues. Meanwhile, I was impressed with the obvious open source successes like Apache and Linux, it became interesting to see if we could develop a commercial-grade product as an open source project.

We were founded in 2001, we are a 501(c)3 non-profit, we have about fifteen staff (employees and volunteers) but we hope and expect commercial enterprises will build a whole ecology of products around Chandler, some open source, some not. As far as our funding goes, I put up the initial money, and we've since received a grant from the Andrew W. Mellon Foundation.

Chandler is an open source Personal Information Manager (PIM), so it will manage all the usual things -- email, calendars, contacts, and tasks as well as free-form items of information like notes. It's intended to be a general information item, so MP3s, blogs, any item of information is fair game for Chandler.

Sharing and collaboration is very important in Chandler on a peer-to-peer, server-optional basis. (Really, Chandler should be called an inter-personal information manager.) It's almost shameful that it's 2003 and it's still much harder for non-technical people than it ought to be. While have some arguments about terminology -- what's a server, what's not -- the bottom line is that non-technical people ought to be able to set up their service without having to go to school to figure it out.

Chandler should run equally well on all three platforms, Windows, Mac, and Linux, and it does that already. It is important to us that nobody get lost.

Chandler is also a platform for developers to build other things on top of and it will be modular and extensible. Andy will talk more about that in a minute.

At the heart of Chandler is a flexible database that is organized the way people think rather than how computers typically store records. One of the things you want that for is so that you can connect and associate, different kinds of items of information.

The first program that I know of that did this was Agenda, which was a PIM that I helped develop fifteen years ago. We at OSAF like to think that Chandler will have "the spirit of Agenda", with a rich ability to associate and interconnect all kinds of documents -- from MP3s to blogs to what-have-you.

Our team puts a premium on the user experience -- we want Chandler to have power and ease of use. One way that we want to give a premium user experience is to make everything customizable and extensible.

If you're interested in more, there's some documents on the Web site, particularly the Compelling Vision document.

Okay, so what do we think Chandler's outstanding features will be?

• We want Chandler to be the power email client of choice. We want to give people better tools for managing large volumes of email quickly and more efficiently, including tools to assist in prioritizing their messages. We've also noticed that many people organize their work lives around their email inbox, and we want to add features that make doing so more practical. We also believe that transparent and robust secure email is important. We know that's a hard problem, but we're committed to it.

• Sharing and collaboration is another key element of Chandler. The mantra is simple: "share anything with anyone". We want to be able to just look into someone else's repository. We also want to be able to subscribe to someone's information so that you can get updates automatically if something changes, and we want to be able to collaborate smoothly on various activities. I don't want to have to tell people ever again when my cellphone number changes, I want to change it in my own repository and have it pushed out to others.

(Andy Hertzfeld speaking)

In addition to Chandler being a terrific application, we also want it to be a great platform that can be easily extended. There's kind of two dimensions for that, extensible for developers and for end-users.

We have a commitment to lowering the bar for scripting by using a graphical front-end for scripting. The hard part of programming has traditionally been keeping the universe of possibilities in their head, but programming is actually pretty constrained. We want to have a graphic front-end to the script so that users don't have to keep in their head all the vocabulary, all the verbs. (Audience interruption: Like Frox? A: Yeah, like Frox, which was a project I worked on a long time ago. I actually think of it more like Hypercard. It's a shame that the state of the art is now 15 years old.) Users should be able to basically select things from menus to write scripts, instead of having to be a programmer.

Another thing that I'm personally very excited about is our agent framework. With the Internet, the world is much vaster than your own machine. There's always lots of stuff that is constantly going on that you'd like to take actions in response to.

We want agents to be able to take actions on their own that orchestrate complex activities, with a gentle notification of when things happen. We're hoping that the agents plus scripting will make it so that you don't think of it as programming as much as just specifying what you want and making it happen.

It should be easy to share customizations as well, using the sharing capabilities built into Chandler. so that even if you aren't a developer yourself, you should be able to shop around for customizations that appeal to you.

There's a whole universe of people out there, doctors, teachers, accountants who have their own way of doing things and who hopefully will make and share their own customizations. We want to take the network effect and apply it to whole populations. So I think that's an exciting part of it.

Naturally, we want to make things easy for developers too. The Chandler framework is very modular, with the main unit of modularity, we call a parcel. Parcels will be able to do anything! They will have access to all Chandler items, including our UI, persistent object database, sharing, security, and so on.

As an open source application, developers have the ultimate flexibility in that you can add any features that you want or to use our subsystems as pieces of other applications.

We're standing on the shoulders of Richard Stallman and Linus Torvalds and all the other great open source pioneers; we hope that some of our subsystems act as the substrate for other projects, so five years from now, other open system projects will be built upon the work we do on Chandler.

So let's look under the hood for a minute. How are we going to do this?

• One of our key goals from the very beginning was for Chandler to run equally well on the Mac and Windows and Linux and potentially others in the future.

  That's hard because it's hard to take advantage of platform-specific features. We looked for a GUI toolkit that we could use to abstract out the UI level. After lots of investigation, we found a framework called wxWindows that we think is excellent and can bear the weight of our expectations. It's quite mature on Windows and Linux and has been making great strides on the Mac in the past year. It's been in development for about ten years now and supports Windows, Mac, and Linux.

  One of the things we liked best is that is uses native widgets whenever possible, so that users will have a consistent UI experience. If you're running OS X, for example, Chandler will look like an OS X application. So wxWindows is a very important part of what we're doing.

• We also decided to develop most of our application code in Python, which is usually considered a scripting language. Python is a high-level language that you can write very clean and elegant code in. We figured that in this day and age, CPU cycles aren't the scarce resource -- developers' time is. Python is so flexible that it lets us iterate very rapidly to leverage the developers' time. Python also makes Chandler available to a broader development community because its easier to master than something like C.

  I'm a python fanatic these days, I find that I'm able to program about three times faster than I could in Java, and I was able to program in Java about three times faster than I could in C.

  Python also has great integration with C, so that if we find that Chandler isn't fast enough, we have the alternative of dropping into C. We expect that to make Chandler fast enough, we'll need to drop some parts down into C.

• Another important decision was to use a persistent object database. Persistence makes it really really easy to do a lot of things that are hard otherwise, things like undo or saving state.

• If we're successful, you're going to have a very intimate connection to Chandler. Your whole life is going to get poured into Chandler. This means that you've got to be able to access it how you want to, like on a PDA.

  To do that, we've carefully separated the UI front-end from our back-end pretty carefully, with a separation between the back-end data side and the front-end viewing side, so that we can eventually have multiple front-ends for PDAs, kiosks, cellphones, in airports, really everywhere. We've developed a protocol called RAP to glue the front-end UI to the back-end database.

  We have more information on our Web site, just go to the link on the sidebar labeled For Developers We are a completely open source project, so if we have documentation lacking, it is just because we didn't have time. Oh, or we because we just haven't had time yet to design it yet.

• And, as I've mentioned before, something that I'm personally excited about is our agent framework. Something the GUIs really haven't done well is orchestrate complex actions. We want to do this with anthropomorphic agents, using metaphors of people in occupational roles, like mailmen or construction workers. This should give a very gentle interaction with the agents.

• We're using RDF as one of our core technologies to provide flexibility to add attributes.

We had some exciting news a few days ago: on Monday we released our 0.1 release. We are now a really open source project, which is great. A lot of open source projects never make it this far. (laughter) You can now download executables on all three platforms and we encourage you to download and check it out.

So let's take a look at Chandler. This isn't exactly our 0.1 release because we've been working on it, but you can get our up-to-date code by checking it out from our CVS repository at http://cvs.osafoundation.org, login anonymous, password anonymous. If you're insane enough, you can track our progress by the hour. We will be making dot releases as well, every few months, that you can download.

We also have an open bugzilla at http://bugzilla.osafoundation.org. One thing that I can promise is that we have a lot of bugs in our first release; you can help us by finding bugs and helping us figure out which ones to fix. (We also use bugzilla for developer tasks so you can see our priorities; we're completely open.) We also have a wiki which is really fertile ground for ideas. We have a number of enthusiastic volunteers who have contributed a lot to the wiki. There's also a vibrant mailing list.

Everyone on the OSAF team is committed to building a best of breed software, but we're nowhere near there. Please don't judge us by our 0.1 release -- this is just a beginning, an embryo that will grow into a beautiful child sometime over the next year or so.

Now, there are some caveats. We are committed to a world-class user experience, but it's not up to our standards yet. We haven't worried a lot about look and feel yet. We only have a smidgen of functionality so far in calendar and contacts, but we don't have email or our security framework in place. We guarantee that it will crash a bunch, so you shouldn't use it for your day-to-day use. This release is just for checking it out and getting the ball rolling.

(fires up Chandler) The first thing to note is the navigation framework. We've taken a page from what we've learned from Web browsers, such that every functional area has its own URL, and in fact so does every data item, and in fact every attribute of every data item. To go to a certain View, you can just enter in the URI for that View. (points out URL box)

One way of navigating is just to type in the URL of the page. One thing that falls out of that decision is that it's easy to have my best friend, the Back button. This makes it a lot less scary to go somewhere because you always know that you can go back. Having URLs everywhere also makes it easy to do a Home view and bookmarks although we don't have those implemented quite yet.

Usually you'll navigate with the sidebar, which lets you switch between different parcels. We'll be shipping with five or six different parcels, but we expect that developers will come up with many more.

As I mentioned before, Chandler is very modular, and this central area (gestures) is all owned by the current parcel. We have a number of parcels (which I'll go into in a minute) for Calendar, Contacts, IM, and a couple of sample parcels.

The parcels all have a framework of multiple views showing you a different slice of the data. There are also different View types, like here in Calendar we have a month View and a Week View.

Okay, let's look at Calendar (CalendarScreenshot). Note that we've only really just scratched the surface on our UI, just to get the ball rolling and to get it out there. It's still a little rough. But here's the calendar, I can drag and drop to make appointments (does) and I can move them around (does). I can change between weeks. One of our key concepts is that you can have multiple Views onto each data set, and here's an example: we can look at the data by week or by month. This is extensible by the developer and eventually we want this to be extensible by the end-user. Hopefully you'll have a lot more functionality soon.

Okay, now here's our Contacts manager (ContactsScreenshot).

You can see the photos of people, here's John Anderson, our system architect. It's structured, where each contact has a number of different contact methods, and each has its own semantics.

I can change the picture (does); I can change the contact semantics, so I can add an contact address (does).

I can also click on a contact address and do something useful, for example if I click on Mitch's Web site then it takes me to his blog, which is a great way to keep up to date with what's going on in our project. Note that the Jabber ID shows that Mitch is present. so if I click on "Jabber address", it will let me send Mitch an instant message using our built-in instant messenger.

I can say "add a contact method", and then add something like Mitch's cell phone number (I won't put his real number in because he'd probably kill me if I did).

When you create a Contact, you can use one of a variety of templates becausethere are different people in your life -- friends, business associates, etc -- and they have different types of contact information.

We also have different ways of showing the data -- you can see Contacts in a table format, or as cards, and we have three different sizes of card.

You can also have different Views of the data, so for example, if I click on "Coworkers" in the sidebar, now only my coworkers show up in the View.

You can also create new Views. So for example, if I want to only see the volunteers associated with Chandler, I can create a new View that only shows the volunteers (I tagged the volunteers earlier). (Creates a new View)

The first release is for developers, and the Repository (RepositoryScreenshot) parcel shows the actual RDF of the repository. It's kind of a worms-eye view of the data. If it's a string or something you can see it directly, and if it's a higher-level construct then you can see its structure. This isn't much use to end-users, but it's a really good way for developers to explore the repository at a deep level.

Remember one of our key goals is to make it easy for people to extend parcels. This parcel, Zao Bao (which means "morning newspaper" in Mandarin) (ZaoBaoScreenshot) was actually written by Chao Lam, our product manager. While he's a good programmer, we didn't hire him to do programming, and he wrote this parcel on nights and weekends even though he'd never written Python before.

The point is that it takes very little time to develop these -- it's a matter of days, not weeks.

We use Jabber as our instant messaging framework in part because it is extremely well designed but also because it's completely open.

We've built a simple little IM client into Chandler, so I can send a message over to Mitch (does). You can see on Mitch's screen, my Roster entry it turns green until Mitch sees the message. (Mitch responds).

The exciting thing is that we can use Jabber not just for instant messaging but also as a framework for all of our sharing stuff. So after this talk, I'm going to be hungry. So I can ask Mitch if he knows any good restaurants around here. (sends IM to Mitch asking about restaurants)

Mitch: What Andy doesn't know is that inside my Contacts I have a bunch of restaurants. I can go to my Restaurants View and you can see Eliza's and Zao Noodle Bar. So now if I click on this little disclosure triangle then I can switch my sharing policy to "public".

Andy: And now you can see that Mitch's Restaurants View pops up in my Roster, and if I click on it, there are Mitch's Restaurants.

You can also see that Mitch is making his calendar available. I can go to his calendar (does) and see Mitch's calendar. I can also choose to overlay my calendar on his (does). You can see my appointments in green and Mitch's in beige.

This is just a taste of what we hope to have in Chandler in a mature way in Chandler 1.0

The biggest opportunity is allowing people to collaborate easily. Because we're using an instant messaging framework for our base data transport layer, we can work around the complexity of setting up servers. Normally, it's hard to reach people over a network because they change IP addresses all the time, they go behind firewalls, and so on. IM handles all those problems, so piggybacking on IM solves a bunch of problems for us.

So that's where Chandler is today, and now Mitch is going to tell you what's ahead.

Mitch speaking

We have a Product Roadmap online on our Web site that I encourage you to look at. That has more details about how we're planning on proceeding.

We are going to have a sequence of dot releases every 60 to 90 days. I expect that in the next week to ten days, we'll post a list of what features we want to have in the 0.2 release. We're hoping that in the next year, the most extreme, brave, developers could use it every day.

Canoga is our "1.0" release that we're hoping to have done next year, and I don't mean December 31st. (By the way, Chandler is named after Raymond Chandler, so our code names are based on cities near LA, where Chandler's novels were set.. so our first release is "Canoga".)

Canoga will be for small-scale organizations which would not have a centralized server.

We were approached by the Andrew W. Mellon foundation to do some investigation. They gave us a grant of about $100,000 to investigate what kinds of things we'd need to do a large-scale organizational development. As a result, we're putting together what we'd need for our "Westwood" release, which is Chandler for higher education. This would have a standards-based calendar, interoperate with campus infrastructure, and support for nomadic access.

So how can you contribute? We're now at a point where there's actually a lot that you can do for us now. We see the community as an integral part of our development process. You can find bugs and tell us about them at http://bugzilla.osafoundation.org. You can submit patches. You can build your own parcels. I encourage everyone to participate in the mailing lists and on the wiki. We've had a lot of great discussions on both the mailing lists and the wiki that have been very productive.

A number of us come from the commercial world, so are really nervous about releasing code that isn't really ready, but we put it out to be a good open source project. We are expecting the unexpected in the same way that putting stuff up on our web site lead to unexpected things like the Mellon grant.

This is just a start, we're hoping that we can make something really really great. Chandler will come into its own in the next few months. This is your project as much as ours.

Any questions?

---

Q: How will you support PDAs?
A: Mitch: To be determined, but we've been very careful to separate the front end from the back end. And this is all simple enough that we hope that if you need a conduit, you'll write a parcel for it.

Q: Is there a Web service available for a back end?
A: Andy: It's very likely that we'll have SOAP APIs to the repository. Right now we use RAP, which is based on BEEP, but it should be reasonably simple to add in a SOAP API.

Q: Is there Undo?
A: Andy: We want to have an undo, but it's not there yet. John Anderson: There are two ways to do it, I think we understand the issue pretty well and we just have to pick one and implement it.

Q: Are the agents are server-side or client-side?
A: Andy: The front-end is on the client side, and we're working on the UI issue on how the user specifies the clients. However, you want the agents to also have a piece on that's close to the data. How we're going to get agents running close to the data is a difficult question and we know that we have to figure that out. We know that to have agents work right, we need to have a strong security framework so that server operators feel comfortable running the code, but we haven't figured that all out yet.

Q: How do you run Jabber on this, I assume you need to get a Jabber client.
A: Mitch and Andy: No, no, Chandler is a Jabber client, all I have to do is get a Jabber ID and go to Edit->Preferences. A lot of Jabber servers, like jabber.org, have open registration, so you they'll register you automatically. One caveat is that you have to turn the karma on your Jabber server up.

Q: What does the competetive landscape look like?
A: Mitch: The good news about open source is that what you see is what you get. There isn't really any competition. This means that today and for the foreseeable future, we're not competitive with anything that is appropriate for large enterprises. At some point as it matures, it might get there but it isn't there yet. In the Canoga timeframe, we're looking for the infocentric user who gets a lot of email

Getting things right for the users is absolutely important and we will not compromise. That's a real luxury that we had to give up in the commercial world.

The long-term success depends upon what kind of ecology can spring up around Chandler. Think about Linux... we hope to develop the same sort of ecology. We won't know if we'll get there or not, but we have to give it a shot. There are other products out there that have tens of millions of copies out there, there's no reason that we can't.

Q: Will your license allow commercial development?
A: Mitchell Baker: Our intention is that we want to make code available publicly and for If you want to do GPL, then can already do that If you want a commercial license, then you can come talk to us and we will work on getting a commercial license.

Mitch: We're not successful if we make commercial development impossible.

Q: It seems that most of the information that I care about is stuff I don't own, stuff I access by reference and not by value.
A: Yes, and Rys McCusker^? has done a lot of work on that. It became apparent that we have information from a lot of different sources and we'll need to do some compositing of data. The canonical example is IMAP, where a student's email may stay on the IMAP server forever, but there might be some metadata that we'd store in Chandler. It's a requirement and we have the beginnings of the architecture for that.

Q: RDF doesn't scale very well. Is it possible to put an X-query across multiple servers?
A: Rys McCusker^?: We've started defining the query language, but haven't defined it yet. Mitch: I think you've hit upon a question that we haven't resolved yet, but if it turns out that we need to do that, then we'll do that. Juergen Botz: We're debating that right now.

Q: Is there a way to vote online [for features]?
A: Yes! There's a public wiki, there's lots and lots on the wiki (more than you want to look at, probably) and you should post what you're interested in there.

Q: I want to emphatically support what Scott just said (about X-queries).
Q: If you have a soft schema model, what's your scripting model for getting at the various attributes.
A: Katie Capps Parlante: We're working on that, but every item has an attribute, and there are links to other attributes. In Python, I'm just accessing attributes in a dictionary. [discussion that got way too deep quickly] Mitch: Katie Capps Parlante and Brian Skinner have written something that talks about our data model and how attributes are accessed; I would give every encouragement to read that because we've not only thought a lot about of this, but because we want you to jump in and give us comments.

Q: Devil's advocate: Evolution has been successful, why do we need Chandler?
A: Andy: Evolution is Linux-only, and we want something cross-platform. Mitch: Also, Evolution is a great product, but it's design center is Outlook, and we want to develop something fundamentally different.

Q: Can you talk about your vision for email?
A: First thing is that you need to be able to process a large volume of email efficiently. You need to be able to do all the things you want with a single click. But also, people use email in a task management type of framework: what do I have to answer, what do I have to deal with, what's junk? We'll do that in a number of ways, but you'll be able to View things by project so that you can see all your tasks, contacts, etc associated with that project. Nothing should be further than a click away. Andy: I'm hoping that we'll have automatic transparent security, so that with zero configuration you'll have secure email.

Q: Will you use email as a transport to share information?
A: Andy: Yes, of course.

Q: I assume you're going to port this to all kinds of different devices, right?
A: Absolutely, cell phones, web front-ends,.... but we're counting on our developer community to do a lot of the ports to all kinds of different things.

Q: Any chance you'll make the back-end database modular?
A: With developer effort, absolutely. A: The data that Chandler manages doesn't have to all live in Chandler. For example, students might want to leave their contacts information on an LDAP service so they can get at it from anywhere. We're planning to do data compositing in a number of places.

Q: Can you talk about your development process?
A: There are at least half-a-dozen people on the staff who live/breathe/eat/sleep software design. We've done some structured interviews. The mailing lists and wiki have had some incredibly interesting discussions. Bill Joy said that most of the smart people in the world don't work for you, and we really believe in that... People have come into the project with visions: Ducky has written two books and has thought about email probably more than anyone else on the planet, Andy has thought about agents for a long time, I've been involved with Agenda, and so on. One of the most interesting things for me is to see all the creative vision that has been coming into the project.

Q: You mentioned that you wanted to seed commercial development.
A: At some point, if this meets with acceptance then it would be logical for some of you to take this into the enterprise. We don't feel that's our core mission, but we want to get to the point where it's meaningful for other people to come in and develop those products.

Success in the open source community is to have an ecology that is a mix of commercial and non-commercial products.

-- DuckySherwood - 25 Apr 2003
-- DennisLynch - 16 May 2003