EIMML Project

The goal of the EIMML project is to create a schema independent format that is used by Chandler to exchange data with other applications. In Chandler Preview, this is used for:

• Sharing through Chandler server (Cosmo): the sharing protocol is now using EIM as the basis of its new Morse Code protocol
• Dump and Reload: the objective is to allow users to migrate their Chandler repository items across schema and application versions.
• Chandler to Chandler communication: EIM is also used to exchange updates of items through emails between different instances of Chandler

The EIM framework uses internal-schema-specific translator callbacks to serialize items into a format that can be read in by future Chandler versions.

---

Architecture

Driver

The "driver" is the code which is told when and where to dump the repository to, and is then in charge of determine which items to dump, and invokes the translator and serializer accordingly, storing the output to a file. Similarly, it can be told to reload a dump file and will invoke the deserializer and translator in order to create items.

Model/Translator

In order to dump/reload items we need to define the EIM schema for each kind of item. Because of work already done in the SharingProject, we have record types defined for most user-visible "Content" items. These EIM "record types" are called the model. The code in charge of converting items to their corresponding EIM records and back again is called the translator. For example, if we want to dump/reload an IMAP account, we'll need to define an IMAP account record type which contains an ordered list of fields such as "host", "username", etc. Then we need to implement translator callbacks that can import an EIM record (and thus create or update an item), or can yield EIM record(s) for an item.

Serializer

The SharingProject currently uses EIMML as a serialization format. It is the serializer's job to take EIM records and serialize them to a format, and also deserialize the format into EIM records.

Initially we were looking at PyTables^? to handle the serilization, but after a quick IRC session between Bear and PJE it was determined that PyTables^? had a more restrictive view on core data types and the work to wrap our data types in PyTables^? would be cumbersome. During that discussion PJE remembered cPickle and now the next step is for me (Bear) to mock up a test to make sure cPickle will handle the requirement.

Startup/Installer

During startup Chandler needs to recognize that an upgrade has occurred and search within the profile directory for the most recent repository and any dump file. If one is found, then the startup process needs to warn the user and start processing the file.

If it's not found, then the user needs to be warned about the lack of a dump file and offer to exit.

The Installers now need to be changed to install into a versioned directory in the same manner as the profile directory.

---

What we want to Dump/Reload

Data

• Items
• Collection membership
• Stamps
• Collections

User Profile

• Account settings

Collection metadata

• What collections were Mine versus Not-Mine
• Collection order
• Collection color

Shared Collection metadata

• Whether a collection was Published versus Subscribed
• What attributes were included/excluded from Sharing cloud
• Tickets: Read-Write and Read-Only
• Free-busy calculations are the same

Preferences

• Use timezones
• Default timezone

View settings (Overkill, for sure)

• What was the user looking at last:
  • Selected App area
  • Selected Collection
• What collections were overlaid
• What Triage sections were opened/closed
• Column sort
• Scrollbar position
• Window size
• Pane size (Sidebar, Mini-cal, Summary, Detail view)
• Use sections (I'm not even sure that this should be optional any more ...)
• The items in the View->Configure Window menu (Of those, the Visible Hours setting would arguably be the most used).

---

Proposed Models

John suggested this for Collection:

    renameable               = schema.One(schema.Boolean, defaultValue = True)
    color                    = schema.One(ColorType)
    iconName                 = schema.One(schema.Text, defaultValue = "")
    iconNameHasClassVariant  = schema.One(schema.Boolean, defaultValue = False)
    colorizeIcon             = schema.One(schema.Boolean, defaultValue = True)
    dontDisplayAsCalendar    = schema.One(schema.Boolean, defaultValue = False)
    outOfTheBoxCollection    = schema.One(schema.Boolean, defaultValue = False)
    canAdd                   = schema.One(schema.Boolean, defaultValue = True)
    allowOverlay             = schema.One(schema.Boolean, defaultValue = True)
    searchMatches            = schema.One(schema.Integer, defaultValue = 0)

---

Project Status

• What works:
  • The Test menu has two menu items for dump/reloading files. Dump will iterate through all content items and serialize whatever EIM records are applicable to each. Reload will import items from a file.

• Done
  • Set up wiki page
  • Dialogs for dump/reload file are hooked up to Test menu Morgen Bug#8142
  • Added osaf.dumpreload module which will dump a list of items using provided translator and serializer, and also reload from file. Morgen
  • Added a simple "pickle" serializer Morgen
  • Serialization format Bear Bug#8141 - decided on cPickle

• In progress
  • Core code to drive dump/reload, including figuring out what items to translate/serialize; question: should we always just iterate every item in the repository? Morgen Bug#8142
  • Startup/Installer code Bear Bug#8143

• Not started
  • Define EIM schema and implement translators for... :
    • Shares Morgen Bug#8144
    • Accounts BKirsch Bug#8145
    • Collections/sidebar John Bug#8146
    • Prefs Jeffrey Bug#8147

Meeting notes : 03/13/2007

We had a meeting on Tuesday March 13th, 2007 to review the work left to be done and how the transition to EIMML will be handled.

List of tasks remaining to complete serialization of all items

Follow up on Morgen's email. For Sharing:

• EIM/Morse code: Can be turned on by creating an "experimental" sharing account via the Chandler account dialog. That's for publish. For subscribe, use the pim URL associated with the read/write ticket. For the moment, there's only one server instance implementing this (under BCM's desk) and only Morgen, BCM and Mikeal should be hitting that server. Dogfooders and others are asked not to start switching to that server, even for testing...
• Email: BKirsch added things but not sure when to commit. BCM mentions that things need to be hashed out on the Cosmo list first.
• Events translator: (Jeffrey, flying to Atlanta today), record fields agreed upon but implementation to be done
• Triage translator: (Stearns) Morgen and Bryan agreed on this over email on the list already.
• Sharing: Coordinate with Cosmo before committing stuff that change sharing. Adding fields to a record should be handled transparently by Cosmo but changes to fields may break stuf. The Cosmo team is on it so shouldn't be broken too long.
• Process for changes: Send a mail to the Cosmo list on what the record type is. Add unit tests for these in the sharing tests (important!)

For Dump and Reload:

• Dump and Reload: There are Test menu items (under Test/Sharing) already, dump is simply an extension of the sharing translator (iterating through all items), need to add the rest we need to translate (accounts, collections, etc...)
• Collection (John): Trying to save as little as possible without saving the whole set, lots of unknown in those tasks. Note that this has no impact on morse code sharing.

Current transition strategy

• See Morgen's page on this
• Could we simply throw all the pure XML stuff (would loose anything other than events)? Could be annoying for dogfooders... What to do? Sheila, Katie and Philippe to think about that (several possibilities to envision)

Switching to EIM entirely

• Chandler will have the ability to have new and old sharing for a while.
• Eventually, we will rip that off for Preview so this needs to happen within a month or so
• When? At least a couple of weeks, may be more. In a recent email, Grant raised issues that are very much unknown at that point. Needs review and comment on the list.

-- MorgenSagen - 15 Feb 2007