March 7th, 2006

Last week I started playing with TurboGears. Its one of a few really slick web frameworks written in Python. There are of course a ton of web frameworks and parts of web frameworks, but TurboGears is one of the recent standouts. It combines a bunch of pre-existing open-source web projects to form a really nice Model-View-Controller style framework like Ruby on Rails.

You get a very clear web framework stack:

1. MochiKit^? - View - An awesome javascript library providing tons of useful tools for making custom html widgets, tweaking the DOM, etc.
2. Kid - View/Controller - a templating language similar to Cheetah. Its big boast is that templates begin as strict XHTML and come out as strict HTML. This is different than Cheetah which takes a preprocessor-style approach to templating. It also means you can edit them pretty easily in existing HTML/XHTML editors like Dreamweaver.
3. CherryPy - Controller - Modela web application server. The basic idea is that URLS come in and map directly to a particular object and method path. Form values come in a through parameters to these functions. Results from these functions are passed out to the Kid template as a dictionary of values
4. SQLObject - Model - an object-based view into an SQL database.

My interest is trying to build chandler by replacing SQLObject with the repository. It turns out the SQLObject/CherryPy binding is probably the weakest link in the whole chain, so at least from a model/controller perspective it was very easy to swap that out.

Installation

I started by installing TurboGears 0.9alpha using the instructions they give on the web site, using Chandler's Python installation. I downloaded ez_setup.py and then used it to download TurboGears:

> release/bin/RunPython.bat ez_setup.py -f http://www.turbogears.org/preview/download/index.html TurboGears

Then I used the tg-admin script to create a new project. I ran into a little hiccup because the tg-admin script doesn't seem to like making a new project inside a directory with stuff checked out from Subversion. So instead I made a new directory and installed it there.

> cd ..
> chandler/release/bin/Scripts/tg-admin-script.py quickstart

I called my project "ChandlerWeb"

Starting the server with the repository

When you start a project, TurboGears gives you a nice little script that fires up an instance of CherryPy on port 8080. Your entrypoint into the webserver is in the form of a series of controllers that get called when a particular url is accessed.

The first thing I had to do was get the repository up and running in this process and accessible from my own controller. I decided to use Morgen's great headless.py script which does all the initialization for you. I had to set CHANDLERHOME appropriately so that I could still find chandler's python and parcel path from RunPython^?.bat.

Then I just needed an import line, and to create the repository:

from tools import headless

mainView = headless.startup(create=True, profileDir='C:\\alecf\\ChandlerWeb')

Then I used RunPython^?.bat to start the server:

../chandler/release/RunPython.bat start-chandlerweb.py

The first issue I ran into was that the reloader scripts that automatically reload TurboGears^? when you change code was really messing with the repository. The reloading system works by partially starting your server, then forking off a new instance of your server. When the code changes, the new instance shuts down and the partially-started server restarts the actual server. The easiest solution was unfortunately just to turn off autoloading in config.py

autoreload.on = False

While I was there I realized I would probably need some session support - to maintain state across sessions. That was pretty easy as well:

session_filter.on = True

Now when the server starts, a repository instance is loaded, and the main view is stored in "mainView"

Getting to the repository

CherryPy does its own management of threads and connections, making sure that everything you're dealing with stays local to your own connection. The way the Repository works, we generally want one view per thread so that different threads don't all stomp on the same view. I needed to hook into CherryPy's thread-creation code to create the view and store it somewhere my controller would find it.

Fortunately, CherryPy provides an easy way to hook into thread creation. You just need to write a function that should be called when a thread is created, and store the view in some thread local storage.

def connect(threadIndex):
    newView = mainView.repository.createView("thread %s" % threadIndex)
    cherrypy.thread_data.repView = newView

cherrypy.server.on_start_thread_list = [connect]

Getting Chandler Data on screen

I mocked up the chandler UI in HTML. I'll leave this as an exercise for the reader.

The first chandler data that we wanted to get out of the repository and up onto the screen is the list of collections in the sidebar. To do this, I used kid's py:for attribute to iterate the items in the sidebar collection. (This is actually a collections of all the collections in the sidebar)

I made my own Chandler controller, and hooked this into the CherryPy controller hierarchy. Then I defined a method, 'index', which would retrieve the sidebarCollection from the repository and return it to kid.

def Chandler(controllers.RootController):
    @turbogears.expose(html=".templates.chandler")
    def index(self):
        view = cherrypy.thread_data.repView
        sidebarCollection = schema.ns("osaf.app", view).sidebarCollection
        return dict(sidebarCollection=sidebarCollection)

cherrypy.server.root.chandler = Chandler()

Then in chandler.kid, I just had to define my for loop to make a UL list:

<div id="sidebar">
  <h2>Sidebar</h2>
  <ul>
    <li py:for="coll in sidebarCollection">
      <a href="#" onclick="loadCollection('${coll.itsPath}')" py:content="coll.displayName"/>
    </li>
  </ul>
</div>

Interactivity

Above notice that I added a call to loadCollection() and passed in the path of the item. I wanted to have the collection get loaded into the summary view without the page loading. I decided to use MochiKid^?'s neat JSON support to load the collection asynchronously.

The real trick here is to make a general-purpose API to get data out of the repository in JSON and let the browser do whatever it wants with that data.

I made a new controller called Repository that would provide this JSON interface.

class Repository(controllers.RootController):
    pass

cherrypy.server.root.repository = Repository()

Then I needed to define a new method to return the contents of a given collection. I called it collectioncontents.

class Repository(controllers.RootController):
    @turbogears.expose(format="json")
    def collectioncontents(self, path=None):
        if path is None:
            raise cherrypy.InternalError()

        view = cherrypy.thread_data.repView
        view.refresh()
        coll = view.findPath(path)

        ...

TurboGears^? provides this great way of returning JSON data rather than rendering a Kid template, by simply adding @turbogears.expose(format="json") as a method decorator.

The rest of the setup here is unfortunately kind of complex. I'll go through each line explaining why we need each part:

        if path is None:
            raise cherrypy.InternalError()

CherryPy is really great about passing parameters out of the url into your methods. For instance, if I visited http://localhost:8080/repository/collectioncontents/foo/bar then it would want to pass 'foo' as the first parameter, 'bar' as the second parameter, etc. The problem is that our repository paths also use '/' as a separator. If we just appended this to the url, it might look like http://localhost:8080/repository/collectioncontents///parcels/foo/bar which would really confuse CherryPy. In order to pass in our parcel path in a friendly way, we have to use CGI-style parameters. This means using named parameters like 'path', which means giving them default arguments in Python. So now the url for visiting a path is going to look like http://localhost:8080/repository/collectioncontents?path=//parcels/osaf/framework/blogs.

I may investigate a way to pass the string in a way that it can be automatically packed on the JS end and unpacked on the Python end.

        view = cherrypy.thread_data.repView
        view.refresh()
        coll = view.findPath(path)

First we have to go retrieve the view, and then call refresh() to make sure we have the latest data from the repository. It would be really nice if view was accessible as part of the controller. I'm wondering if it might be possible to store thread-local data in the controller, so I'll be investigating that. It would also be nice to have refresh() called automatically so we don't put it at the start of every call.

Finally, we retrieve the actually collection from the repository.

Next up, we need to retrieve the items in the collection and return them to the user. For reasons I won't go into here, its easiest to just create an array, and return them as part of the json result, rather than trying to return the collection directly.

    def collectioncontents(...):
        ...
        items = []
        for item in coll:
            itemdata = {}
            for attr, value in item.iterAttributeValues():
                itemdata[attr] = unicode(value)

            itemdata['who'] = getattr(item, 'who', "")
            itemdata['about'] = unicode(getattr(item, 'about', ""))
            itemdata['date'] = getattr(item, 'date', "")
            items.append(itemdata)

            
        return {'result' : items }

On the frontend, we need to write some JavaScript to retrieve this data, in the function loadCollection(). We'll write some simple JavaScript:

function loadCollection(path) {
  path = escape(path)
  d=loadJSONDoc("/repository/collectioncontents?path="+ path + "");
  d.addCallback(loadCollectionDone);
}

This will start the asynchronous call back to TurboGears^?, and call the local function loadCollectionDone when complete:

function loadCollectionDone(result) {
    result = result.result;
    values = [];

    rows = []      
    for (itemIndex in result) {
        columnNames = ['about', 'who', 'date', 'triageStatus'];
        cells = []
        for (colname in columnNames) {
            colname = columnNames[colname];
            value = result[itemIndex][colname];
            if (value instanceof Array)
                value = value.join(', ');

            cells.push(TD(null, value));
        }
  
        rows.push(TR(null, cells))
    }

    replaceChildNodes("summary-tbody", rows)
}

The above code makes a bunch of table rows (using MochiKit.DOM's globals like TR and TD which automatically construct their respective DOM nodes) The table is defined so that it has a TBODY which is easily replaced:

    <div id="summary">
      <table id="summary-table">
   <thead>
     <tr><th>about</th><th>who</th><th>date</th><th>triage</th></tr>
   </thead>
   <tbody id="summary-tbody"/>
      </table>
    </div>

April 28, 2006

I hadn't touched TurboGears in a while but I noticed that they've released their alpha4 release and I decided to upgrade turbo-chandler to use that. I brought everything up to TurboGears^? alpha4, but there was a bug in the kid template egg that came from the cheeseshop. It seems it assumes there is an encoding though TurboGears^?/TurboKid may not always give it an encoding. So I had to make this change for now:

--- kid-0.9.1-py2.4.egg/kid/pull.py~    2006-04-28 09:49:25.508406400 -0700
+++ kid-0.9.1-py2.4.egg/kid/pull.py     2006-04-28 10:54:22.572108800 -0700
@@ -187,8 +187,10 @@
     if not isinstance(value, str):
         value = str(value)

-    return unicode(value, encoding)
+    if encoding is None:
+        return unicode(value)

+    return unicode(value, encoding)

 def _coalesce(stream, encoding, extended=1):
     """Coalesces TEXT events and namespace events.

I managed to make TurboChandler into a parcel after making the above change.