Instrumentation

Adam Bosworth provides a good explanation of how Web servers use instrumentation to quickly respond to user preferences.

• "Services provide all actions as URL's and Button clicks and these can be statistically analyzed by services to better understand what's working for customers and what isn't. Thus as thus statistical instrumentation highlights customers needs, the services can instantly and dynamically respond to these customer needs. In short there isn't some person who is the "boss" of a service. The customers are, very literally, in control. Think of it as democracy versus totalitarianism. This last point showed up almost from day one in the web. The user interface customized itself to the users needs, location, and data in a dynamic way through the magic of dynamic page layout. Today, a full ten years later, most windows apps still don't do that. But heck they are only 2 or 3 or 4 generations evolved. Services, in the last decade, may have evolved 600 times by now all in reaction to what they have learned directly from customer use."

Chandler may not be able to evolve as quickly as a service but there's no reason we can't have just as sophisticated instrumentation. A simple instrumentation framework (something to keep track of event counts, stopwatch timing, and event sequences) would do wonders for learning how people spend their time and use Chandler, and ought to be in place by the time we do usability testing.

Instrumentation is like unit testing, you do bits here and there as you go along. But before we can even start to have instrumentation we need a bit of framework, and that framework needs to be in the schedule one of these days. A high-level proposal:

• Have a run-time flag to turn instrumentation on or off (like debug logging can be on or off)
• Store instrumentation data in a file, so that we can get at it independently. E.g. users can give us their instrumentation data without revealing the contents of their data.
• Load instrumentation data into memory on startup if instrumentation is on.
• Have a simple instrumentation API with methods like "incrementCount(eventName)"
• Store the instrumentation data to file periodically and on shutdown.

-- LisaDusseault - 05 Nov 2004