Chandler Automated Test System 0.2 (CATS 0.2)

Chandler Automated Test System (CATS) is a test framework built to write and execute python test cases for Chandler. In June 2006 the framework was rewritten to be more modular, less error prone, give more granular feedback, and to begin the transition to OAF (Open Automation Framework), a new multi-product test framework in development at OSAF. For information on the old framework you can look at ChandlerAutomatedTestSystem.

The new framework has many important features.

• Test scripts are written as classes which inherit from ChandlerTestCase (see WritingChandlerAutomatedTestsWithCATSZeroPointTwo)
• Framework contains concept of state (Suite, Test, Action) and encapsulates reports (PASS/FAIL calls) in actions which then bubble up their results. This allows very granular error report detection.
• Built in traceback captures.
• Improved test output. In addition can output to stdout and stderr.

Where CATS lives.

CATS is contained in the chandler/trunk/ repository in the chandler/tools/cats directory.

svn co svn+ssh://svn.osafoundation.org/svn/chandler/trunk chandler

Running CATS Tests.

You must be in the chandler directory to run these tests. To run the entire suite you can use.

./release/RunChandler --create --scriptFile=tools/cats/Functional/FunctionalTestSuite.py

./release/RunChandler --create --chandlerTests=TestNewEvent:TestNewEvent

./release/RunChandler --create --chandlerTests=TestNewEvent

./release/RunChandler --create --chandlerTests=TestNewEvent,TestNewMail,TestNewMulti:TestNewMulti,TestCalView

Creating a log file

--chandlerTestLogfile=fileName.log.

If you want a log file you will probably want to set DEBUG and MASK values as well (see below). The log file will be created in chandlers profile directory. Instructions on finding the profile directory are here. If a log file of the same name already exists the existing file will be renamed to be prepended with the date/time of its creation. The most recent log file will not start with the date/time stamp.

DEBUG and MASK parameters Control test logging output

• debug:
  • 0 = show only failures and use easy read format (masking ignored)
  • 1 = show pass and fail
  • 2 = show pass and fail and check repository after each test
• mask:
  • 0 = show everything
  • 1 = don't show report
  • 2 = don't show report, action
  • 3 = don't show report, action, test

Debug value is passed to chandler with the -D or --chandlerTestDebug options Mask value is passed to chandler with the -M or --chandlerTestMask options

• examples:
• ./release/RunChandler --create --chandlerTests=TestNewEvent --chandlerTestMask=0 --chandlerTestDebug=1
• ./release/RunChandler --create --chandlerTests=TestNewEvent -M0 -D1

Output (default)

The default values of debug=0 and mask=3 produce the output shown below. This sample shows two failures the first one causing a traceback. Tracebacks will always be printed twice, once when they happen and a second time in the test report at the end. The report is followed by a *Test Result* summary and then finally values for tinderbox to read.

09:14:37_<chandler> tools/do_tests.sh -f
Using [/home/Dan/chandler] as the chandler/ directory
- + - + - + - + - + - + - + - + - + - + - + - + - + - + - +
Started Fri Jul 21 09:15:15 PST 2006
Setting up script environment
Running functional tests
Running C:\cygwin\home\Dan\chandler\tools\cats\Functional\FunctionalTestSuite.py
- - - - - - - - - - - - - - - - - - - - - - - - - -
C:\cygwin\home\Dan\chandler\tools\cats\Functional\FunctionalTestSuite.py []
Traceback (most recent call last):                                
  File "C:\cygwin\home\Dan\chandler\tools\cats\framework\runTests.py", line 51, in run_tests
    test.runTest()
  File "C:\cygwin\home\Dan\chandler\tools\cats\framework\ChandlerTestCase.py", line 68, in runTest
    self.startTest()
  File "C:\cygwin\home\Dan\chandler\tools\cats\Functional\TestNewEvent.py", line 135, in startTest
    EditCalendarStartDate=(True, evtThirdDate),
  File "C:\cygwin\home\Dan\chandler\tools\cats\framework\ChandlerTestLib.py", line 876, in CheckDisplayedValues
    self.CheckBlockVisibility(blockName, visValueTuple[0])
  File "C:\cygwin\home\Dan\chandler\tools\cats\framework\ChandlerTestLib.py", line 774, in CheckBlockVisibility
    block = getattr(App_ns, blockName)
  File "C:\cygwin\home\Dan\chandler\parcels\osaf\framework\scripting\proxy.py", line 191, in __getattr__
    return getattr(self.app_ns, attr)
  File "C:\cygwin\home\Dan\chandler\application\schema.py", line 1106, in __getattr__
    raise AttributeError(
AttributeError: HeadlineBlock is not in <module 'osaf.app' from 'C:\cygwin\home\Dan\chandler\parcels\osaf\app\__init__.pyc'> or <P
arcel: app ced160f0-18fc-11db-8f09-00904bb16a50>

SUITE ChandlerTestSuite FAILED         
        TEST TestNewEvent FAILED None
Test Failure due to traceback         
Traceback (most recent call last):
  File "C:\cygwin\home\Dan\chandler\tools\cats\framework\runTests.py", line 51, in run_tests
    test.runTest()
  File "C:\cygwin\home\Dan\chandler\tools\cats\framework\ChandlerTestCase.py", line 68, in runTest
    self.startTest()
  File "C:\cygwin\home\Dan\chandler\tools\cats\Functional\TestNewEvent.py", line 135, in startTest
    EditCalendarStartDate=(True, evtThirdDate),
  File "C:\cygwin\home\Dan\chandler\tools\cats\framework\ChandlerTestLib.py", line 876, in CheckDisplayedValues
    self.CheckBlockVisibility(blockName, visValueTuple[0])
  File "C:\cygwin\home\Dan\chandler\tools\cats\framework\ChandlerTestLib.py", line 774, in CheckBlockVisibility
    block = getattr(App_ns, blockName)
  File "C:\cygwin\home\Dan\chandler\parcels\osaf\framework\scripting\proxy.py", line 191, in __getattr__
    return getattr(self.app_ns, attr)
  File "C:\cygwin\home\Dan\chandler\application\schema.py", line 1106, in __getattr__
    raise AttributeError(
AttributeError: HeadlineBlock is not in <module 'osaf.app' from 'C:\cygwin\home\Dan\chandler\parcels\osaf\app\__init__.pyc'> or <P
arcel: app ced160f0-18fc-11db-8f09-00904bb16a50>

                ACTION Traceback detected Action created by framework for report call
                        REPORT Action created by framework for report call
        TEST TestEditModeOnEnter FAILED 
                ACTION Check_DetailView
                        REPORT CheckEditableBlock (On display name Checking)  || detail view value = New Event ; expected value = ⁿYour title here\u062b

*********** TEST RESULTS ************  
*************************************
TestCreateAccounts____________ passed
TestAllDayEvent_______________ passed
TestNewCollection_____________ passed
TestNewEvent__________________*FAILED*
TestNewMail___________________ passed
TestNewTask___________________ passed
TestNewNote___________________ passed
TestTableSelection____________ passed
TestStamping__________________ passed
TestSharing___________________ passed
TestMoveToTrash_______________ passed
TestDeleteCollection__________ passed
TestMulti_____________________ passed
TestCalView___________________ passed
TestRecurrenceImporting_______ passed
TestRecurringEvent____________ passed
TestSwitchingViews____________ passed
TestExporting_________________ passed
TestFlickr____________________ passed
TestImporting_________________ passed
TestImportOverwrite___________ passed
TestCertstoreView_____________ passed
TestEditModeOnEnter___________*FAILED*
TestEventStacking_____________ passed
TestBlocks____________________ passed
*************************************
#TINDERBOX# Testname = ChandlerTestSuite          
#TINDERBOX# Time elapsed = 0:06:56.880000 (seconds)
#TINDERBOX# Status = FAILED

Output using other values

Using values of debug=1 and mask=0 would produce output similar to what is below. This output could be further modified as follows:

• A mask value of 1 would suppress the lines beginning with Report,
  
• A mask value of 2 would suppress lines beginning with Report or Action
  
• A mask value of 3 would suppress lines beginning with Report, Action or Test
  

*** Test Report ***
*Suite ""ChandlerTestSuite"" Failed :: Total Time ""0:05:09.088459"" :: Comment ""None""
**Test ""TestNewEvent"" Failed :: Total Time ""0:00:05.709902"" :: Comment ""None
None""
***Action ""Test recurrence dialog"" Failed :: Total Time ""0:00:00.037924"" :: Comment ""None""
****Report ""Didn't see the recurrence dialog when deleting a recurrence instance"" Failed :: Comment ""None""
***Action ""CheckEditableBlock"" Failed :: Total Time ""0:00:00.000209"" :: Comment ""(On EditCalendarStartDate Checking) 
|| detail view value = 6/30/06 ; expected value = 7/1/06""
****Report ""(On EditCalendarStartDate Checking)  || detail view value = 6/30/06 ; expected value = 7/1/06"" Failed :: Comment ""None""
**Test ""TestTableSelection"" Failed :: Total Time ""0:00:33.123805"" :: Comment ""None
None""
***Action ""Check_Equality"" Failed :: Total Time ""0:00:00.000403"" :: Comment ""Table Selection Ranges || [(1, 1), (3,
4), (9, 9), (11, 11)] != [(11, 11)]""
****Report ""Table Selection Ranges || [(1, 1), (3, 4), (9, 9), (11, 11)] != [(11, 11)]"" Failed :: Comment ""None""

As you can see the state level is exposed by the amount of leading characters, Suite(1), Test(2), Action(3), Report(4). For successful and non-PASS/FAIL output the '+' character is used as the leading character. Failures use '-', but notice that since this stage of the output is in real time, the failures have not bubbled up to cause failur in the action, test, and suite they were contained in.

After the test suite has run a test report is printed which shows the finalized results, with failures analyzed and the results bubbled up through their encapsulation. Example: After the failure report is printed a one line summary is output which is intended to be both easily readable and easily parsable.

$Suites run=1, pass=0, fail=1 :: Tests run=24, pass=22, fail=2 :: Actions run=358, pass=351, fail=7 :: Reports run=547,
pass=540, fail=7