Developer notes from Morgen

This document is the result of a couple of conversations I had with Morgen near the time of the OSAF transition. It’s mostly a detailed brain-dump of the sharing code, but also some notes about other code he had written.

Dialogs

The Accounts dialog has been worked on a fair deal by bkirsch.

Subscribe & Publish dialogs are similar to in approach to Accounts (i.e. loading panel out of xrc).

The Unsubscribed Shares dialog, like many recent dialogs, does not use xrc (i.e. all its widget/sizer setup is done directly in python. This is because of i18n concerns.

The Proxies dialogs uses a bunch of dynamic resizing code that came from RobinDunn.

The Activity Viewer (Can be found in the Tools->Sharing menu): It was never particularly publicized.

• If you want to show progress: Create an Activity object, pass it in to your background process, and that can call the activity’s .update() method to change progress.
• .update() will raise an error if the user has aborted the Activity.
• A Listener (like a dialog) can listen to a single Activity, or to all activities (like the Activity Viewer dialog).
• Listeners are currently responsible for making sure events get posted in the wx thread.

Gdata Parcel

gdata sync was close, but it looked as if recurrence was going to be hard (they represented it differently) -- Jeffrey knows more. One direction was hard, and had to do with the way they model event exceptions in recurring events, so maybe a solution would be to download in .ics and upload in gdata (assuming I got the direction right).

Sharing Code

The code for the can be found in osaf.pim.sharing parcel (svn).

Modules well-known to other people:

• webdav_conduit.py - Grant
• ootb.py - Grant
• ChooseFormat.py - Jeffrey
• eim.py - pje
• ics.py - Jeffrey
• simplegeneric.py - pje
• symbols.py - pje
• WebDAV.py - Grant

__init__.py

• Background sync handler (line 179 onwards): shutting down in particular is complex; see comments

• PeriodicTask calls run() ~ line 288
  • handles sync manager stuff
  • altView business is so that each share sync can be self-contained
• stats2str() produces an abbreviated log of what the get & put did for each sync. Format is similar to the output in chandler.log (e.g. "<<", "--", "!!").

• publish() is simpler than it used to be (delegate) to account.
• You can also "publish" any share in-memory for debugging purposes (in Tools menu).

• subscribe()
  • inspect() figures out what's going on with the remote URL (i.e. does it support CalDAV/!MorseCode etc).
  • 'priv:write' determination might not work in all cases (funky servers)
  • The format attribute (and associated classes) now gone: instead of conduit/share/format it's now conduit/share (+ translator + serializer).
  • Conduits can be used with/without an account. In the latter case, the conduit stores user/password/host, etc; else it gets pulled from the account (there's a helper fn for that).

• unsubscribe()
  • ignoreCollection on CosmoAccount is for the case where you unsubscribe and we don't want the sync manager to turn around and ask you about subscribing to this collection.

• '# Formatters for conflicts' section: these are used for localized strings that end up in the conflict dialog.

• Could nuke getFilteredCollectionDisplayName(); it’s no longer used.

shares.py

When you share a collection, the collection gets a !SharedItem stamp and its 'shares' points to all the shares that collection appears in. Individual items in the collection get 'sharedIn'. For example, if you add an item to a shared collection, and it hasn't been synced yet, that item's sharedIn won't include the share. This mechanism could support private items in a collection and also per-item shares.

If I do an edit/update and I send an item over mail, I get a peerState attached to the item, and for each recipient I get a peerState with the peer being the recipient's email address. So, we use the existing peerState when a recipient sends back an update (used to be problem with mail address matching including full name, which was wrong).

What is a State? It represents the relationship between a remote and local item. There is one per share or per peer. An item can have multiple States if it has been shared in multiple ways. We persist the agreed state (which is the EIM representation of what has been agreed by both parties) and pending (incoming changes that are conflicts). Internally, these are stored as pickles, and PJE wrote the pickling code (so questions to him :).

merge(): PJE and Morgen worked pretty closely on how this all works. Code is relatively self-documenting. Also, code will automatically ("just-in-time") turn log level to logging.info if it encounters any conflicts.

autoResolve(): There are two levels of auto-resolution. One can happen at the EIM level, which is what is implemented here. This works by calling the translator's resolve() method, which is where you can implement model-specific stuff (like triage status resolution for ex). Return value is -1, 0, 1 (for pick first, leave as conflict, pick second). There's a second level which is done in recordset_conduit.

updateConflicts(): Needs to be called whenever pending is changed, so that insternal conflict-related state can be made consistent.

Conflict class: The conflict dialog works with Conflict objects, and will get one for each pending change on the item. It calls things like .apply() on each, which in turn makes sure that things like updateConflicts() have been called. The .verify() method actually makes sure that the given Conflict really still applies (for instance, it could have been changed by the user, or been synced by another share, etc).

Share class:

• hidden might not be used or not.
• format could probably be removed.
• conflictingStates could be used to show whether a given collection has conflicts (currently, we show collection errors in the sidebar, but this makes it possible to distinguish conflicts from errors).
• filterClasses isn't currently used, but might be needed for CalDAV (when a server doesn't support VTODO).
• Cloud declarations can be removed: we don't use them any more. Same for .fileStyle().

Share.sync(): a TokenMismatch is what's generated when someone else syncs in the middle of your sync. This works by the server telling you a sync token when you get(). Then, we pass that token back to the server in put(), and the server will tell us (205 - someone else did a put since you got the token; or 423 - the collection has been locked -- i.e. PUT is in progress). Either of these causes a TokenMismatch to be raised. We try at most 3 times till we give up.

Share.reset(): needed to get around problem where server gets upset about deleting a record that's not there. Jeffrey and Morgen found another case where this would happen: a commit() failed at the end of a sync, and so Chandler is forced to "lose" the fact that it deleted a record. So, they are thinking the server shouldn't barf (maybe just warn).

Share.isAttributeModifiable(): in any share you have, the sharing layer allows you to make local changes anywhere; the UI enforces readonly-ness of attributes. M has a @todo in conduits.py to make this work.

recordset_conduit.py

Logic for retrieving inbound changes, figuring out what has changed locally, calling merge() to figure out what to apply/send.

_sync() became really huge because of recurrence; Jeffrey is familiar with some of its workings.

alias is the whole UUID or UUID:recurrence-ID business for recurrence.

getRecords()=/=putRecords() are what concrete subclasses implement to do serialization. (i.e. these are overridden by {Monolithic,Diff,Resource}RecordSetConduit. There are further subclasses, e.g. for Cosmo, or WebDAV.

displayName: there were cases where Shares didn't set their displayName, but subsequent code depends on it. There's some code at lines 137f, 216f that messes with displayName. The latter trickily tries to make sure that remote collection renames only propagate if the user hasn't locally changed the name.

Bunch of recurrence stuff by Jeffrey.

cosmo.py

Some changes (e.g. to sync manager attributes) must be changed in the sharing thread or else the repo will end up having conflicts.

Might be some duplication with init.py's subscribeMorseCode().

CosmoConduit.getFilter(): each filter can apply to multiple fields. There are a bunch of filters that we apply for Cosmo; there are a bunch of fields that are there for dump & reload, but that we exclude from MC sharing. This allows us to have a single sharing/dump+reload model, and single translator, etc.

callbacks.py

Callback stuff that pre-dates Activity class

Application.py registers itself with sharing for unsubscribed collections, so the "restore shares" dialog gets shown as needed.

Similarly, Application.py registers for updates for the app status bar.

itemcentric.py

Stuff for email-based sharing

inbound() actually returns a list of "similar"-looking email addresses (i.e. ones with case variations or different full names). This allows inbound() to match the peer (email address) this was originally shared under.

line 109: (Jeffrey knows more). Trying to solve: you have a recurring series with a "pure" occurrence, and a modification to that occurrence comes in, that modification will appear to be in conflict (because the local item has a bunch of values via inheritFrom). So, this code conses up a record that has a bunch of eim.Inherit fields, so as to match what's really going on here.

We don't currently allow emailing deletions to other people.

stateless.py

Using sharing to do import/export to files without a share.

We used to have OneTimeFileShare(?) that would be an item that enabled them.

serialize.py

Can't remember if it's used any more ... Maybe .ics uses it?

utility.py:

getOldestVersion(): used by Andi to figure out how far back we need to keep repository versions, so we can compact. Unclear what would happen if you compacted to a more recent version: is this just an efficiency thing? (In the dump&reload case, we don't save the repo version of last sync).

localChanges(): not used any more

serializeLiteral(): only used by P2P plugin

getExistingResources(): only used by WebDAV account publishing -- to find a unique name.

ootb.py

The Cosmo account that gets created "out of the box" when Chandler starts up.

eimml.py

RecordSet → xml translation (Can get rid of the EIMMLSerializerLite class)

tests/round_trip.py

When debugging (e.g. debugging a unit test failure), you can do share.sync(debug=True) to turn on debug logging for one particular sync (e.g. to debug a test failure).

tests/xTestCosmoSharing.py

This runs the current round_trip.py test against a cosmo server (you need to add login in). It isn't part of the normal test suite (i.e. has been disabled with the 'x' prefix) because it needs to be run from headless.py right now, in order to run with the twisted reactor.

-- GrantBaillie - 17 Jan 2008