QA and Release Engineering meeting - Monday August 14th 2006

Agenda and Notes

CATS status and roll-out plan

• New framework CATS 0.2 has been dropped in for Chandler. The Tinderboxes has been turned on for 1 cycle and turned off.
• We are not running the perf tests with CATS 0.2 yet, only the functional tests.
• We need 2 days of work to get that done.
• Bear suggests to turn the -N switch around to turn the new tests by default
• ETA: next week, needs to be done before alpha4 Code Complete
• Action: Dan working on this

Disabled tests status

• We have 4 functional tests and 3 perf tests disabled. What's the plan to turn them on again?
• Dan to work on turning on the functional tests after CATS 0.2 functional tests switch finished but before perf tests.
• Action: Heikki to look at the perf tests, Dan the functional tests

Tinderbox monitoring

• Who's doing what? When? Dan and Heikki hasve been doing a lot of it.
• We can officially split it between Heikki, Dan and Bear.
• Mikeal proposes that some tests we know are failing randomly to not generate a notification so that people are paying attention to them. Heikki is saying to ping him for those tests (mostly the Flickr and Amazon failures).
• Also, there's a link to the error log that points to the wrong log. Bear is aware of that.
• Action: Bear to put the monitoring people on the "always notify" list, also write a script so that only the on duty person is notified.

Performance tests

• Apparently, the Linux box needs to be rebooted to keep the perf good...
• Action: Monitoring persons should also monitor the perf. Just needs to be once at the end of the day. Check that there's no persistent "red".

wx plans

• Bear coordinates with Robin. No auto merging done yet but no need to do that yet. Bear to check with Robin when we can turn that on.
• Bear in contact with the people handling the wx CVS tree. Will talk to them about tests, etc...
• They have Tinderboxes but apparently not testing the GUI functionalities.
• We don't have new hardware for the wx Tinderboxes. We tried to use VMWare to get that going but had issues with them (clock's wrong, mouse issue when using VNC,...). Consequence: we are no running the Tinderboxes on the wx builds.
• Action: Heikki and Bear working on which VM solution we'll be using in the end (VMWare, Xen or Parallel)

Cosmo/Scooby?

• Merge under way. Need to write new tests. QA will need 2 weeks to make sure that whole merge project is stable.
• What about the Apple tests? Great but no framework to run them. They also implemented lots of stuff we haven't. We'll try to run them against Cosmo in the next month or so. We'll have failures since there's a lot of stuff we haven't implemented. We'll add the Apple server to the list of officially supported server.
• Mikeal to push the tools out today.
• Selenium: Adam working on how we could use it. Bunch of issues (no support for DnD, issues with dynamic div IDs. etc...).
• OAF to be code complete this week. Cosmo will be the first customer of it.
• Action: Mikeal and Adam working on the test suite for the merged project in the next 2 weeks.

Chandler milestones order in Bugzilla

• Issue with the order of milestones in Bugzilla
• Action: Philippe and Heikki to solve that

-- PhilippeBossut - 14 Aug 2006