Running Chandler Tests

Nowadays, testing is handled by the rt.py python script (in the tools/ subdirectory. You can just launch the script directly - it will figure out how to execute Chandler with the correct version of python.

• Use tools/rt.py --help to see an overview of all rt.py options
• If you’re interested in finding out the exact command rt.py will execute for any test (or tests), use the --dryrun command-line option.
• rt.py will always run a Localization check as the first test
• When running any GUI tests, be sure that your screensaver doesn't kick in - this can cause some false failures.
• Interacting with Chandler while tests are running can cause false failures. Also, on most platforms (Mac & Linux?), Chandler must be the frontmost window, and must be left alone while those tests run. On Linux you can use the xrt.sh wrapper to run tests in nested XServer to work around this issue.
• To debug failures running rt using wing, you can setup wing for remote debugging and use --params='--wing' when running rt to stop at breakpoints and debug problems. Also, in Chandler, you can run recorded scripts manually from the test menu, completely avoiding rt, which is often the easiest way to debug individual failures.

Unit Tests

• tools/rt.py -u

Functional Tests (and Recorded Tests)

• tools/rt.py -f

Recorded Tests

• If you're running recorded scripts on Mac you will need to set System Preferences > Keyboard and Mouse > Keyboard Shortcuts, Full keyboard access to "All controls". This is required because the tests were recorded on windows and windows allows focus to be set for all controls. By default Mac doesn't allow that. To avoid recording different scripts for each platform we're changing the default for mac.
• Recorded scripts currently use asserts for verification, so make sure you are running non-optimized Python otherwise verifications won't happen
• tools/rt.py -r

More info on recorded scripts here

Performance Tests

The current focus is mentioned on PerformanceProject.

• see the busy developers guide to Chandler optimization see how to profile the performance tests.
• on Mac: rt.py times startup performance using "gtime", but GNU Time isn't installed by default - get the source from http://directory.fsf.org/time.html, build it, and install the resulting "time" binary somewhere on your path (eg, "/usr/local/bin/gtime"). (You could also get it from darwinports.)
• The quick display format while rt.py is running will display the last timed test in the file, if there are several tests in a single file. So for example in the PerfLargeDataSharing.py file the quick display will show the value for Subscribe test only. If you let the script run to completion, all the results will be printed.

Running all tests in one go

• ./tools/rt.py -p

The startup time test with empty repository

• ./tools/rt.py startup

Tests that do not have LargeData in the name (i.e. tests with empty repository)

• ./tools/rt.py <script name>

Tests that have LargeData in the name (i.e. tests with large repository)

The large repository is created by the PerfImportCalendar test, which is usually run at the start of the performance suite. So, you can start out by:

• rm -fr ./test_profile
• ./tools/rt.py PerfImportCalendar # creates the large repository backup

Now, running each of the tests can be done like this:

• ./tools/rt.py <script name>

e.g.

• ./tools/rt.py PerfLargeDataNewCalendar

You will need to re-perform the above steps (removing the test_profile subdirectory) if you have updated your Chandler to a different schema version.

The startup time test with large repository

• You must first create the large repository backup, see above.
• ./tools/rt.py startup_large

Repeating tests (and getting median)

The --repeat option is handy if you want to see if anomalous results are due to random variation or not. For example,

• ./tools/rt.py -t PerfNewEventFileMenu --repeat=10

will run the PerfNewEventFileMenu test 10 times in a row, throw out the high and low data point, and tell you the median and standard deviation in the mean.

Running single tests

• tools/rt.py -t NameOfTest

Tinderbox runs

See http://builds.osafoundation.org/tinderbox/Chandler/status.html for testing status.

See what tests each box is running here: ChandlerTinderboxes