Perf Data Comparisons

We run Chandler performance tests using artificial test data sets. The main data is a 3000 event ical file we load for the so called "large data" sets. For scrolling test we also load two small ical files (under 10 items total) and for sharing test we use an ical file which has about 10 items. For some time there has been questions about how accurately this test data set matches real world performance with real world data sets.

I finally did a comparison using Katie real world data on Linux. Katie subscribes to the office calendar, and has a home, work and book club calendar. When restoring the settings Chandler asks if timezones should be turned on - I answered yes.

I also compared with Ted's data. Ted had work and office calendar, and I also turned on timezones during restore.

When we run the traditional large data tests the repository starts with 8840 items in the repository. Katie's repository contains 5377 items (at the time I run this test). Ted's repository contains 6286 items.

Below is the comparisons of the results. Some notes: sharing test was disabled with katie's and ted's data to prevent data corruption; table scrolling test failed with katie's and ted's data (not enough items); switch to all view, jump week, overlay and switch calendar measured only the 2nd time of the action.

The cases which give the most concern are those that show negative difference of more than 10%. This means the real world case is significantly slower than what one would expect from our test results.

There are some caveats: the test results could change significantly based on what collections we are using for the tests, as well as how much data is visible on the week that tests are performed. The tests jump to a fixed date to get consistent results, but in the real world your performance will vary.

How to run the tests yourself

• svn up -r 13291
  • You could try some other revision as well, but this is what I tested
• make distclean install purge
• Shut down all processes you can. You should have just a terminal window open.
• ./tools/do_tests.sh -p
  • We are interested in the top section of the output only, i.e. something like:

PerfImport.py [ 60.0s ]
...
Timing startup with large repository 4.62 4.85 4.76 [4.76s]
Showing performance log in 5 seconds, Ctrl+C to stop tests

• • You can hit Ctrl+c at that point. Run the test again. You now have your baseline numbers.
• patch -p4 < katie.diff
  • katie.diff
• Now you need to edit tools/do_tests.sh: find "katie_profile" and change it to something like "yourname_profile".
• Then you need to edit tools/QATestScripts/Performance/PerfLargeData*.py files: change "Home" and "Office_Calendar" to some collection names in your sidebar - the tests assume that "Office_Calendar" has many events (I will later refer to "Home" but use the name of your collection).
• Make sure PerfLargeDataSharing^?.py does not exist - if it does, delete it.
• ./chandler --profileDir="yourname_profile"
• Test > Restore settings...
• Select the "Home" collection in the sidebar
• File > Save Changes
• File > Backup... into "yourname_profile" directory, quit Chandler
• ./tools/do_tests.sh -p
  • Like above, you can hit Ctrl+c at the point. Run the test again. You now have your comparison numbers (some irrelevant tests were now omitted compared to the unedited do_tests.sh).
• ./release/RunPython tools/headless.py --restore="yourname_profile/__repository__.001" --profileDir="yourname_profile"
• view.check()
• len(view)
  • this tells you how many items are in youe repository
• Send email to heikki with your results. Include your platform, svn revision number, both baseline numbers, both results numbers, mention how many collections you have in the sidebar, how many items in your repo, and whether or not you have timezones enabled.

-- HeikkiToivonen - 27 Feb 2007