Transition to EIM

Introduction

External Information Model: EIM is a very elemental representation of information. Once information is in EIM form we are able to operate on it in a semantic-neutral fashion. We can diff, merge, detect conflicts, and serialize it without understanding its underlying meaning.

Essentially EIM is an object-relational mapping framework. In it you define record types (which are akin to RDBMS tables), translators (which convert items into records and back again), and serializers (which convert records into XML and back, etc.)

EIM also facilitates schema evolution, because it makes it easier for parcel developers to write upgrade code -- they simply write code that translates a more-or-less fixed external format into their parcel's current schema.

We currently have two translators (one for sharing, and another for dump/reload which extends the sharing translator).

We currently have three serializers: one called EIMML for sharing with Cosmo, an EIMML "lite" version that is less verbose, and a Python-pickle serializer that dump/reload will use.

Testing

Inside the sharing package we have doctests and regular unit tests for the core eim module, translator, serializer, the merge algorithm including conflict detection, the conflict resolution API, and the edit/update API.

Doctests in the sharing package:

• EIM.txt : core eim module functionality
• EIMML.txt : serialization, inmemory conduit, merging, filtering, conflicts, edit/update API,
• Translator.txt : translator

Unit tests in sharing/tests:

• TestItemSharing^?.py : more edit/update API tests
• TestMultipleShares^?.py : ensuring an item can be shared in multiple shares

There are special test conduits that act as a mock Cosmo in-memory, either in "resource" mode (non-diffing, like a DAV server) or in "diff" mode (like over the morsecode protocol). Those conduits are tested against a "round trip" script that acts as two Chandler clients sharing a collection, adding/removing items, stamping/unstamping, making non-overlapping and overlapping changes, etc. This same script can be run against a real Cosmo conduit which I use to ensure Chandler/Cosmo interoperability.

"Functional" Round Trip test suite (these all make use of sharing/tests/round_trip.py with various conduits):

• TestEIMDiffRecordSetInMemory^?.py
• TestEIMDiffRecordSetInMemoryLite^?.py
• TestEIMResourceRecordSetInMemory^?.py
• TestEIMResourceRecordSetInMemoryLite^?.py
• xTestEIMDiffRecordSetCosmo.py : actual live testing against Cosmo with Morsecode and EIMML

Transition

Support for traditional dual-fork shares (ICS+XML) and Morsecode are both in the chandler trunk. Morsecode is used if the user chooses an "Experimental" sharing account type in the Chandler account dialog. One possible way to transition an existing Chandler user:

1. Ensure the existing Chandler user has their collections up to date on osaf.us
2. Quit old Chandler and run new version
3. Create two sharing accounts in the Chandler account dialog: a traditional /cosmo/dav account, and a Morsecode /cosmo account
4. For each collection they own:
   • Subscribe to it
   • Unsubscribe (which leaves the collection in their Chandler repository)
   • Republish it using the Morsecode account
5. Delete their old account from Chandler

How much we automate these steps depends on how much time we have.

Remaining Work

• Event translator
• Mail translator
• Ticket creation/retrieval via Morsecode

In addition...

• CalDAV atop EIM

Documents

• SharingProject
• Record Types