Summary of planned improvements to tinderbox and functional tests

This page summarises decisions made at the release/build mtg based on input from emails and the previous meeting. This is intended to clearly state what work needs to be done. Items that already have bugs and are in process are not listed.

Previous meetings & emails on this topic

• Build & release mtg notes
• Dan's email & whiteboard notes
• Bear's email

Accept Bryan's proposal for dealing with tinderboxes when they have not been green for a week or more:

1. P1 bugs get filed for each failure
2. The failing tests get disabled on platforms on which they fail(with bug numbers in the comments)
3. People are assigned to fix those bugs and re-enable the tests

Note: For all reliably failing/100% failures, we will still apply the "roll back" policy.

Improvements to chandlerTestLib (formerly QAUITestAppLib) & functional tests

1. Rewrite ChandlerTestLib so that:
   • most of the validation is done automatically by the framework. (QA to write proposal)
   • contains more asserts to check itself (QA to write proposal)
2. Remove dependancies between tests (QA to write bug on this)
3. Regularly run tests outside of suite (need faster hardware for this not to take too long)

Logging improvements

1. One log should contain output from chandler, twisted and the tests in chronological order (need to log a bug on this)
2. Removing false links from the top of the on line logs (Heikki & Stearns to work on this)
3. Failing tests should print a line showing how test was run to aid dev's in reproducing problem (need to log a bug)
4. Failing test should raise so the traceback could aid locating the problem (start discussion on dev)
5. Change default settings to the test logger for more output (start discussion on dev)

Scripting improvements

1. There is general agreement that modifying scripting to remove the 'magic' app_ns would make understanding test failures easier. John seem to be the only one likely to be efficient at making these changes. It is uncertain if John will have time to make these changes anytime soon.