-- MikealRogers - 05 Jun 2006

Purpose

The new CATS framework was created for the purpose of stabalizing the current automated functional tests and simplifying the process of creating new tests.

This document mostly describes the architecture of framework and isn't intended to be a detailed guide on how to write tests in the new framework.

Differences

The main difference between the new framework and the old is that tests are no longer executable scripts that are fed to chandler, read by a file object and executed as python code. Instead each test is a class, containing a startTest() function.

The framework will instantiate this class, run the method, and move on to the next if one exists.

Another large difference is that the framework is responsible for instantiting the logger object, but the test writer is still responsible for passing the logger to the test library functions.

Besides these major differences the only other changes have been file renames and a few new features, along with an entirely new output object to replace QALogger.

tools/QAUITestAppLib has been replaced with tools/cats/framework/ChandlerTestLib , and has been modified to call the new methods in the output object.

Syntax differences

ChandlerTestCase^?

You must now create a test class that inherits from ChandlerTestCase^?

from tools.cats.framework.ChandlerTestCase import ChandlerTestCase

class MyNewTestClass(ChandlerTestCase):

    def startTest(self):
        #your test code goes here

Lib Import

This:

import tools.QAUITestAppLib as QAUITestAppLib

has become this:

import tools.cats.framework.ChandlerTestLib as QAUITestAppLib

Logger reference

This:

QAUITestAppLib.UITestView(logger)

has become this:

QAUITestAppLib.UITestView(self.logger)

New logger methods

There are many improvements in the new logger, you no longer need to worry about the logger state or any of that nonsense.

To report a pass or fail simple bass True or False to self.logger.report()

self.logger.report(True)

The report method also has a few nice features

self.logger.report(True, name="name of event that just passed", comment="a comment string")

If you wish to time an action, or you wish to encapsulate a series of pass/fail report calls you can use.

self.logger.startAction(name="Series of report calls for for a given set of events")
self.report(True)
self.report(False, name="failed event")
self.logger.endAction()

Running tests

The functional suite is still run with chandler --scriptFile option

./release/RunChandler --create --scriptFile tools/cats/Functional/FunctionalTestSuite.py

Individual functional tests must be in the tools/cats/Functional directory and can be run like so

./release/RunChandler --create --chandlerTest file:test
example:
./release/RunChandler --create --chandlerTest TestSharing:TestSharing
or if the file and test class share the same name you can just use
./release/RunChandler --create --chandlerTest TestSharing

The new output object will print to stdout and a logfile, but only if one is specified.

./release/RunChandler --create --chandlerTest TestSharing --logFile test.log

a timestamp can be appended to the logfile name where the @ character is found.

./release/RunChandler --create --chandlerTest TestSharing --logFile test@.log

Unresolved issues

Although the list of issues may grow after the proposal is sent to the list the only open issue I can recall is:

Currently there is no Chandler code that relies on anything in tools. Since the --chandlerTest option runs a method in the framework we are violating the current trend.

We can either a) Put some of the framework functions in another area of chandler or b) leave it like this and allow a policy that "testing" options can rely on libraries in tools.

To Do List

• Finish logfile name stuff
• Finish parsing and summary code
• Fix do_tests
• Fix performance tests
• Get working in tinderbox after all other issues are resolved