Chandler Quality Expectations

Motivations and Goals

To maintain an efficient development process, we need to maintain an ongoing level of quality from checkpoint to checkpoint and milestone to milestone.

Continuous Builds Run Green

OSAF runs Tinderbox builds continuously, including a suite of automated tests. The expectation is that the build remains working, and that the automated tests "run green". Most developers do not run the tests on every platform before every checkin, so a checkin may occasionally break something on one of the platforms. It is the developer's responsibility to watch Tinderbox after a checkin and fix any such problems. If problems are not fixed quickly, Bear will back out the offending change.

Acceptance Tests and Regressions

We have a suite of tests that cover the major functional areas of Chandler. While we'd like to automate as many of these as we can, some tests are currently done in person. Any bug that breaks the current suite of acceptance tests is considered a regression. If the regression significantly breaks functionality it is considered a major regression, and a high priority bug to fix. Major regressions are considered high priority bugs at any point in the development cycle, because they tend to block other people from doing their work. They can mask other bugs, prevent areas of the application from being developed or tested, etc. An example of a major regression: broken import/export. An example of a minor regression: the wrong font is picked on a particular platform.

Checkpoint Builds, Acceptance Tests and Regressions

We create Checkpoint builds every Monday. Early in the week, Aparna runs the acceptance tests against the Checkpoint build, and logs any failures as 'regression' bugs. This practice gives us a chance to discover regressions on an ongoing basis. As we notice regressions on a regular basis, and consider it a high priority to fix them, we believe that we will be able to maintain a low number of regressions as a matter of habit. While we are striving to reduce regressions, we make no promise that a particular checkpoint build will be free of regressions.

Milestone Builds, Acceptance Tests and Regressions

We target new features for milestones, and create a special build for each Milestone. We create a branch for the milestone. As with the checkpoints, Aparna runs the acceptance tests against the milestone build and notes regressions. Unlike the checkpoints, we will actually hold the milestone until regression bugs are fixed. We expect developers to fix any regressions on the branch as well as in the main trunk.

At the milestone, we also take note of the new features since the last milestone, and update the acceptance tests to cover the new features.

Milestones and Servers

OSAF will maintain servers that work with a particular milestone during the development of the milestone, until the completion of the next milestone. For example, if the m2 milestone is targeted to work with the 0.2 version of Cosmo, and the m3 milestone requires a bug fix or upgrade in Cosmo that breaks compatibility with m2, we will maintain a compatible server for m2 until m3 is completed and accepted. In this case, we would need to maintain two servers, one for the most recent milestone and one for the milestone under development.

In other words, OSAF will always provide a working milestone/server combination that can be used at any time.

Demoability

If someone wants to give a demo of Chandler, they have three options:

• The best option is to use the most recent milestone. The acceptance tests give a good representation of what can be reliably demonstrated. We will have servers up that work against the most recent milestone.

• If you need to demo features that are currently under development, the second best option is to plan to use the upcoming milestone.

• If the timing of the next milestone doesn't work for the demo, we'll need to create a branch for the demo. This option is the most costly, as it requires the overhead of a milestone just for the demo.

The first option requires no special work from the team, although it is always helpful to give people a heads up in case something is amiss. The latter two options require these steps:

• Write up a script of the demo actions, at least 2 weeks before the demo. We need this even if we are targeting a milestone instead of a special branch, so that we make sure the milestone has the new features working properly.
• The QA team will start running the script and entering bugs, so we can track the work carefully.
• Demo bugs need to be considered high priority for the developer.
• If not targeted at a milestone, we branch for the demo a few days before the demo. All demo bugs are fixed on both the demo branch as well as the main trunk.