Chandler Sharing Framework Extensibility
This is obsolete. New document coming soon.
The sharing framework allows repository items to be transported in and out of Chandler in various serialization formats and across various transport mechanisms. This document explains how to write your own item serialization and transport mechanism classes so Chandler can interoperate with other clients and servers.
An attached script contains the example code.
Sharing Framework Overview
The sharing framework shares an Item Collection by instantiating three objects: a Share, a Format, and a Conduit. The Share object maintains metadata about what is being shared (including filtering information). The Format handles serialization/deserialization of items; there are currently two Format implementations: 'CloudXML' and iCalendar. The base Conduit class contains the logic for determining what items need to be imported or exported, and specialized subclasses actually perform the work of moving serialized items into and out of Chandler. There are currently four Conduit implementations: FileSystem, Simple (read-only) HTTP, WebDAV, and CalDAV.
Since Chandler sharing is centered on WebDAV/CalDAV, there are helper methods (sharing.publish( ) and sharing.subscribe( ) ) which detect server configuration and construct the proper set of sharing objects. For example, when sharing via a vanilla WebDAV server, Chandler will use a WebDAV Conduit and CloudXML Format, while sharing via Cosmo requires two Share objects, a CalDAV Conduit and CalDAV Format, a WebDAV Conduit and CloudXML Format. You can construct any combination of these objects to suit your needs.
When a Share is to be synchronized, its Conduit will perform a 'get' operation (all new/modified external items are fetched) followed by a 'put' (all local modifications are published), although you can set a Share's 'mode' attribute to 'get' if you only want to fetch changes and not publish them. The base Conduit class has implementations for _get( ) and _put( ) that determine which external items need fetching and which local items need publishing. Conduit developers will need to implement _getItem( ), _putItem( ), and _deleteItem( ) to actually transfer serialized items in and out of Chandler; _getResourceList( ) to provide a list of external resources; getLocation( ) to return a string representing the share's location (such as a URL or file system path); and exists( ), create( ), and destroy( ) for construction and tear-down of the share. When the Conduit needs to actually fetch or publish an item, it asks the Format object to do the transformation to/from a serialized representation; Format developers will need to implement importProcess( ) and exportProcess( ) to do the transformations, which we'll cover first, followed by implementing a Conduit.
Implementing a new Format
Chandler currently ships with two serialization implementations: one which uses XML to represent an item (plus related items, known as an item's 'cloud') which is referred to as 'CloudXML', and the other implements the ICalendar standard. Developers can add their own serialization implementations by implementing a new subclass of the sharing.ImportExportFormat class.
Required Methods
Subclasses must implement the following:
Returns either sharing.ImportExportFormat.STYLE_SINGLE or STYLE_DIRECTORY
There are two styles of Formats: those which store a collection of items in a single resource (such as a monolithic .ics ICalendar file), and those which store each item as a separate resource within a directory (such as a CalDAV collection where each event is stored in its own resource/file). The fileStyle( ) method should return which style of Format the class implements, so that the Conduit will know whether to pass in an entire collection for that Format to serialize, or to pass in individual items.
Args:
- 'item' is a Chandler item to be serialized
Returns an extension to use when it's time for the Conduit to publish a serialized item.
This method gives the Format a chance to let the Conduit know what extension (such as "xml" or "ics") to append to a resource's name when publishing. Don't include the period in the return value.
Required Args:
- 'text' is the serialized representation of an item to import.
Optional Args:
- 'item', if provided, indicates that rather than creating a new item during import of the text, all attribute assignments in the text should instead be applied to the given item.
- 'changes' contains all changes made locally since the last sync; it is a dictionary whose keys are item UUIDs and whose values are lists of attribute names.
- 'previousView' is a repository view whose version is set back to the time of last sync.
- 'updateCallback' is a method which may be called to provide the user with feedback during long operations.
Obsolete Arg:
- 'extension' is obsolete and will be removed in the next release.
Returns an item.
The importProcess( ) method transforms a body of text into a Chandler item (or items). 'text' is in whatever representation this Format implements, such as ICalendar. It is the responsibility of this method to create new items, if necessary, or to modify existing items. If the 'item' argument is provided, then the Format should use that item regardless of what UUID information is in the text. It is up to the Format to determine whether the text being imported corresponds to an existing item or not. The simplest way to do this is to encode an item's UUID into the serialization (as CloudXML does), and use the repository's findUUID( ) method to look up an item. The ICalendar Format takes a different approach, using the ICalendar UID (as opposed to repository UUID) to look up existing calendar events. This method can make use of the repository's APIs for making attribute assignments, but note that if you want to create an item with a specific UUID, you need to use Kind.instantiateItem( ).
The 'changes' and 'previousView' arguments are there for the benefit of the importValue( ) method in the Sharing module; the CloudXML Format uses importValue( ) to selectively assign literal values such that collisions are detected, and so local modifications are not needlessly overwritten. If your Format class is not using importValue( ) -- which is currently an experimental method that may go away -- you may ignore 'changes' and 'previousView'.
In order to keep the user informed about what's going on, there is a callback mechanism allowing the sharing framework to notify the dialog box code of work being performed. The 'updateCallback' argument is a method you can call to let the higher layers know what's going on. The keywords this method accepts are:
- msg, a message to present to the user
- totalWork, an integer representing how many 'units' of work need to be performed to complete this operation
- work, a boolean, if True it indicates a unit of work was performed
Args:
- 'item' is the item to serialize
Returns the serialized representation of the item.
Optional Methods
The base ImportExportFormat? class provides default implementations for these:
Args:
- 'item' is a Chandler item
Returns an extension to use when it's time for the Conduit to publish a serialized item.
This method gives the Format a chance to let the Conduit know what extension (such as "xml" or "ics") to append to a resource's name when publishing. Don't include the period in the return value.
Args:
- 'item' is a Chandler item
Returns True if the item is one the Format handles, False otherwise.
This method gives the Format a chance to accept/reject an item for serialization. For example, the CalDAV Format only accepts Calendar Event items.
A Minimal Format Implementation: Tab Delimited Format
To illustrate the basics, let's write a simple Format class which serializes Chandler items to tab-delimited text, where each line represents an attribute assignment -- an attribute name followed by a tab followed by an attribute value. We'll also include UUIDs because that's the easiest way to look up an item. For example:
body Some example text createdOn 2005-12-12 18:23:07.109197 displayName Note 1 UUID 65a33d52-6b7f-11da-c20d-be286d830a20
The code below defines a TabFormat class which subclasses from sharing.ImportExportFormat. The fileStyle( ) method indicates this Format treats a Chandler collection as a directory of files, rather than a single file. The extension( ) method informs the Conduit that the individual files should be published with the '.tab' extension in the name. acceptsItem( ) allows any item which is of the Note class (or subclass) to be transformed.
To keep the example simple, our exportProcess( ) is just going to export the displayName, body, and createdOn attributes, converting them to strings before returning the serialized text. Note that the body attribute is a Lob type, and needs additional work to be converted to/from binary format. importProcess( ) does the opposite of exportProcess( ) -- it reads the text, looks up the UUID and creates a new item if one with that UUID doesn't exist, converts the strings to attribute values, and makes the assignments. Note that instantiateItem( ) is a method of the Kind class, and therefore you have to first get the Kind via getKind( ).
from application import Utility, Globals
from osaf import pim, sharing
from repository.util.Lob import Lob
from chandlerdb.util.c import UUID
class TabFormat(sharing.ImportExportFormat):
def fileStyle(self):
return self.STYLE_DIRECTORY
def extension(self, item):
return 'tab'
def acceptsItem(self, item):
return isinstance(item, pim.Note)
def exportProcess(self, item):
values = {
'UUID' : item.itsUUID,
'displayName' : item.displayName,
'body' : item.getAttributeValue('body').getInputStream().read(),
'createdOn' : str(item.createdOn),
}
text = ''
for (attrName, attrValue) in values.iteritems():
text = text + '%s\t%s\n' % (attrName, attrValue)
return text
def importProcess(self, text, item=None, extension=None, changes=None,
previousView=None, updateCallback=None):
view = self.itsView
values = { }
for line in text.split('\n'):
if line:
(attrName, attrValue) = line.split('\t')
values[attrName] = attrValue
if item is None:
uuid = UUID(values['UUID'])
item = view.findUUID(uuid)
if item is None:
parent = self.findPath("//userdata")
item = pim.Note.getKind(view).instantiateItem(None, parent,
uuid, withInitialValues=True)
item.displayName = values['displayName']
lob = item.getAttributeAspect('body', 'type').makeValue(None)
lobStream = lob.getOutputStream()
lobStream.write(values['body'])
lobStream.close()
item.body = lob
item.createdOn = item.getAttributeAspect('createdOn',
'type').makeValue(values['createdOn'])
return item
Using the New Format
Now that we've implemented a new Format, we can put it to work by creating a collection containing some items, defining a Share, a FileSystemConduit, and our TabFormat, and then calling share.sync( ). The following code may be appended to the above module:
def main():
Globals.options = Utility.initOptions()
Utility.initLogging(Globals.options)
Globals.options.ramdb = True
view = Utility.initRepository('', Globals.options, True)
coll = pim.InclusionExclusionCollection(view=view,
displayName='Test Collection').setup()
item = pim.Note(view=view, displayName='Note 1')
attrType = item.getAttributeAspect('body', 'type')
item.body = attrType.makeValue('The body')
coll.add(item)
item = pim.Note(view=view, displayName='Note 2')
item.body = attrType.makeValue('Another body')
coll.add(item)
format = TabFormat(view=view)
conduit = sharing.FileSystemConduit(view=view, sharePath='.',
shareName='tab_test_share')
share = sharing.Share(view=view, contents=coll, conduit=conduit,
format=format)
if not share.exists():
share.create()
share.sync()
if _ _ name _ _ == '__main__':
main()
The first four lines of main( ) parse the command line options, set up logging, and create a RAM repository. Next we create a collection and add two Note items. We create an instance of our TabFormat class, a FileSystemConduit which is configured to publish to a directory named 'tab_test_share' within the current directory, and a Share which has our collection as its contents. Running the script creates two .tab files (one per Note item) within a 'tab_test_share' subdirectory. For an example illustrating how two Chandler instances can synchronize a collection via a Share see parcels/osaf/sharing/tests/TestFileSystemSharing.py
Implementing a new Conduit
While it's a Format's responsibility to transform Chandler items into a serialized form, Conduits actually transfer the data in and out of Chandler. Implementing a Conduit requires providing the following methods:
Returns a string uniquely representing the location of the share's external data
The string returned by getLocation( ) should be in a form suitable for passing to another of the required methods: _getResourceList( ). For a filesystem conduit, a directory path would be appropriate, while a WebDAV conduit would use a URL for its location string.
Returns True if the share already exists in the place this conduit stores data, False otherwise.
This method gives the conduit a chance to prepare the external data store for publishing items to it. For example, a filesystem conduit might create a directory, or a WebDAV conduit would make a MKCOL request.
Remove the share from the conduit's external data store (rmdir, for example).
Args:
- 'location' is a string of the form returned by getLocation( )
Returns a dictionary indicating which items appear in the external data store at the given location. The dictionary's keys are strings representing the 'path' of each item, while the values are 'modification indicators' (see next paragraph).
When the sharing framework needs to find out which items have already been published, it calls this method. The conduit should examine the remote data store and build up a dictionary with the results. For each published resource, the dictionary should contain a key which uniquely identifies the resource within the external data source. For example, a filesystem conduit which stores resources as files would use the filename as the key. The values of this dictionary contain a piece of data which helps to determine whether an external resource has been modified; we'll refer to these values as 'modification indicators'. For example, a WebDAV conduit would use each resource's ETAG, while a filesystem conduit could use modification time. Other conduits (as in our sample below) could use a version number as its modification indicator.
Args:
- 'item' is the Chandler item to publish
Returns 'modification indicator', or None if the Format indicates this particular item is not one the Format supports.
This method is responsible for asking the Format to serialize the item, and then actually publishing the serialized representation to the external data store. If the Format returns None, that is an indicator that this item is not one the Format supports; in this case this method should return None. Otherwise, this method should perform whatever publishing operation it needs to -- write a file, make an HTTP PUT request, etc. Upon completion, the method should return a 'modication indicator' (as described in the previous method) such as an ETAG returned from a WebDAV server, a modification timestamp from the filesystem, or an incrementing version number. This value will be used by the sharing framework's bookkeeping system.
Required Args:
- 'itemPath' is a string which uniquely identifies a resource within the
external data store.
Optional Args (all of which should simply be passed on to the Format's importProcess( ) method):
- 'item', if provided, indicates that rather than creating a new item during import of the text, all attribute assignments in the text should instead be applied to the given item.
- 'changes' contains all changes made locally since the last sync; it is a dictionary whose keys are item UUIDs and whose values are lists of attribute names.
- 'previousView' is a repository view whose version is set back to the time of last sync.
- 'updateCallback' is a method which may be called to provide the user with feedback during long operations.
Returns a tuple: (item, modification indicator)
This method fetches an external resource (uniquely identified within the share by 'itemPath'), asks the Format to convert from serialized form to a Chandler item, and returns a tuple containing the item and the appropriate modification indicator value -- ETAG, modification timestamp, version number -- as described above in the _getResourceList( ) method.
Args:
- 'itemPath' is a string which uniquely identifies a resource within the
external data store.
This method should perform the appropriate operation to remove the specified resource from the external data store.
Optional Methods
The base ShareConduit? class provides default implementations for these:
Args:
- 'item' is a Chandler item
Returns a string that uniquely identifies a resource within the share.
The default implementation (in ShareConduit?) uses an item's UUID in combination with the extension returned by the Format in order to compute an item's 'path'. This could be overridden by a method which uses some other naming convention.
A Minimal Conduit Implementation: Pickle File Conduit
Again, we'll implement a very basic example. This time our external store will be a single file containing a pickled dictionary whose keys represent a share; the values are themselves dictionaries containing published resources (keyed by itemPath). Or in pseudo-python:
data = {
'<share name>' : {
'<item path>' : ( <version number>, <serialized item> ),
'<item path>' : ( <version number>, <serialized item> ),
},
'<share name>' : {
'<item path>' : ( <version number>, <serialized item> ),
'<item path>' : ( <version number>, <serialized item> ),
'<item path>' : ( <version number>, <serialized item> ),
},
}
'item path' is the string that uniquely identifies an item within a share, and is the return value of _getItemPath( ) which defaults to an item's UUID appended with the associated Format's extension. 'version number' will simply be a number that increments whenever that resource is published. The following example includes two additional methods that will probably get added to the list of methods the sharing framework invokes: open( ) and close( ). Until then, if your conduit requires such open/close operations, you will have to call those methods yourself.
class PickleConduit(sharing.ShareConduit):
def __init__(self, name=None, parent=None, kind=None, view=None,
sharePath=None, shareName=None):
super(PickleConduit, self).__init__(name, parent, kind, view)
self.sharePath = sharePath # The path to the pickle file
self.shareName = shareName # The name of share within file
self.data = None
def open(self):
self.data = { }
if os.path.exists(self.sharePath):
dataFile = open(self.sharePath)
self.data = cPickle.load(dataFile)
dataFile.close()
def close(self):
dataFile = open(self.sharePath, 'w')
cPickle.dump(self.data, dataFile)
dataFile.close()
def getLocation(self):
return self.shareName
def exists(self):
return self.data.has_key(self.shareName)
def create(self):
super(PickleConduit, self).create()
if self.exists():
raise sharing.AlreadyExists(_(u"Share already exists"))
style = self.share.format.fileStyle()
if style == sharing.ImportExportFormat.STYLE_DIRECTORY:
self.data[self.shareName] = { }
# Nothing to do if style is SINGLE
def destroy(self):
super(PickleConduit, self).destroy()
if not self.exists():
raise NotFound(_(u"Share does not exist"))
del self.data[self.shareName]
def _getResourceList(self, location):
fileList = { }
style = self.share.format.fileStyle()
if style == sharing.ImportExportFormat.STYLE_DIRECTORY:
for (key, val) in self.data[self.shareName].iteritems():
fileList[key] = { 'data' : val[0] }
return fileList
def _putItem(self, item):
path = self._getItemPath(item)
try:
text = self.share.format.exportProcess(item)
except Exception, e:
logging.exception(e)
raise TransformationFailed(_(u"Transformation error: see chandler.log"))
if text is None:
return None
if self.data[self.shareName].has_key(path):
etag = self.data[self.shareName][path][0]
etag += 1
else:
etag = 0
self.data[self.shareName][path] = (etag, text)
return etag
def _getItem(self, itemPath, into=None, changes=None, previousView=None,
updateCallback=None):
view = self.itsView
text = self.data[self.shareName][itemPath][1]
try:
item = self.share.format.importProcess(text,
item=into, changes=changes, previousView=previousView,
updateCallback=updateCallback)
except Exception, e:
logging.exception(e)
raise TransformationFailed(_(u"Transformation error: see chandler.log"))
return (item, self.data[self.shareName][itemPath][0])
def _deleteItem(self, itemPath):
if self.data[self.shareName].has_key(itemPath):
del self.data[self.shareName][itemPath]
Synchronization Example
Now that we've implemented a new Format and Conduit, let's use them together in a script that simulates two Chandler instances synchronizing a collection. The script below creates two repository views, creates and populates a collection within one of them, publishes that collection via our new Format and Conduit, and then syncs that collection into the second repository view. Changes made in the second view are then synced back to the first view.
def main():
Globals.options = Utility.initOptions()
Utility.initLogging(Globals.options)
Globals.options.ramdb = True
# Set up two repository views (simulating two Chandler instances)
num_views = 2
views = []
for i in xrange(num_views):
view = Utility.initRepository('', Globals.options, True)
view.name = "test_view_%d" % i
views.append(view)
# Create a collection and populate it with two Notes
coll0 = pim.InclusionExclusionCollection(view=views[0],
displayName='Test Collection').setup()
item = pim.Note(view=views[0], displayName='Note 1')
attrType = item.getAttributeAspect('body', 'type')
item.body = attrType.makeValue('The body')
coll0.add(item)
item = pim.Note(view=views[0], displayName='Note 2')
item.body = attrType.makeValue('Another body')
coll0.add(item)
# Display the contents of the collection
print "Collection 0:"
for item in coll0:
print item.displayName, item.itsUUID
# Set up the format/conduit/share to publish the collection using
# our TabFormat and PickleConduit
format0 = TabFormat(view=views[0])
conduit0 = PickleConduit(view=views[0], sharePath='pickleFile',
shareName='test')
share0 = sharing.Share(view=views[0], contents=coll0, conduit=conduit0,
format=format0)
# Publish the share
conduit0.open()
if share0.exists():
share0.destroy()
share0.create()
share0.sync()
conduit0.close()
# In our 'other' repository view, set up another format/conduit/share
format1 = TabFormat(view=views[1])
conduit1 = PickleConduit(view=views[1], sharePath='pickleFile',
shareName='test')
share1 = sharing.Share(view=views[1], conduit=conduit1, format=format1)
# Sync
conduit1.open()
share1.sync()
conduit1.close()
# Now coll1 contains the items that coll0 does
coll1 = share1.contents
print "Collection 1:"
for item in coll1:
print item.displayName, item.itsUUID
# Change an item in coll1:
item.displayName = "Modified"
uuid = item.itsUUID
# Sync coll1 with the pickleFile
conduit1.open()
share1.sync()
conduit1.close()
# Sync coll0 with the pickleFile
conduit0.open()
share0.sync()
conduit0.close()
# The changed item is now changed in coll0 as well
item = views[0].findUUID(uuid)
print item.displayName
if _ _ name _ _ == '__main__':
main()
Show attachments
Hide attachments
| I | Attachment | Action | Size | Date | Who | Comment |
|---|---|---|---|---|---|---|
 txt | samples.py.txt | manage | 7.3 K | 19 Dec 2005 - 13:38 | MorgenSagen | Sample Code |
Log In
See list of page contributors for attributions.