Chandler Desktop Performance Evaluation - IRC Collaborative Session

Objectives

• Review the perceived and measured performance of the Chandler Desktop as it is experienced by a user
• Get a feeling of which scenarios are really problematic and need more work before Preview
• Review the baseline "ideal and acceptable" numbers we're using in the performance matrix so that they are realistic and truly reflect the end user experience and needs

Method

• Which data set? We'll be using the test data set. This Chandler dump file contains a variety of data sent by dogfooders.
• Which server? All aspect of the performances involving a Cosmo server will be measured against the hub.chandlerproject.org server, hosting Cosmo 0.6.1
• Which machines? We need to test as wide a range of machines as possible. Use "native" run (not virtualized) so to be as close as possible to end user experience.
• Who will test? All dogfooders should participate since those are the people who really have a feeling on how things should feel in "normal" usage. All other help welcome of course
• What kind of tests?
  • Primary and secondary scenarios: Those are directly extracted from the the performance project page, we however simplified the list since we're all be working with a big data set. Also, we do not have a plain English description of these scenarios but they should be quite self explanatory.
  • Pet scenarios: We added some scenarios we know are likely to be problematic (e.g. DnD of events on the calendar canvas). We're also open to things you (as a dogfood user) struggle with that are not listed in there. Let us know about it prior or during the test session.
• How to measure?
  • Stop watch: We're trying to get a perceived performance number, between the moment you click till the moment a) you regain control of the app b) the action is effectively done (from your standpoint). Those 2 numbers can be different since some actions do happen in a different thread than the UI thread. If you have enough Python competence, you can add timer into the code.
  • Measure several actions in a row: To get a better number with a stopwatch, it's better to measure something in the 10 seconds range or more. If the unit action you're trying to measure is too fast, try a sequence of identical actions (e.g. if you're measuring scroll, click 20 times in a row in the down arrow and measure the overall time)
  • Measure several times: we need to get a sample of measured values (minimum 3), the more the better, especially if you see significant fluctuation from one test to the next.
  • Share your list of numbers on IRC: One of us (Philippe) will gather the data in a spreadsheet and will update the table here under after the session. Don't just give the average or median value, give all the values you measured. Also mentions if the performance passed your own subjective "acceptable" bar not withstanding what the current perf table is saying (this is why we didn't put those values here under so not to distract or influence you...).
• Where to report the test results? During the IRC session, one of us (likely Philippe) will collate the results and update a spreadsheet. We will use those values to run stats and compare with the current acceptable targets.
• Who's running the session? Heikki and Philippe will be hosting the session.

Before you start testing

Before starting to test you should:

• Download the test data set^?
• Save your own data: launch your old Chandler and use your favorite method (save settings, dump to file, backup respository, save to .ics...) to save your data. After the test, you may want to return to your old trusted Chandler version, it's up to you
• Download the latest 0.7 checkpoint build. You can also run from the svn trunk if you feel like building from source.
• Launch Chandler and reload (File >> Reload items from file...) the test data set
• Set up your accounts: the test data come with no email or sharing account (or we would collide with each other when testing...) so you need to set up using your usual test ground for both. If you really don't have an account suitable for testing or don't want to use one of your own account, you can try one the QA accounts.
• Make a backup of the reloaded repository (Tools >> Repository >> Backup...): in case an unrecoverable bug arise, it'll be faster than reloading from the dump file
• Quit Chandler and relaunch: this is so you start testing as a user would, reloading moves lots of objects in memory so performance measured then are not typical.
• Get on our IRC #chandler channel for chat and directions

Test Cases

• PerfDataSet002.dump: Perf Data Set

Primary Scenarios

id	Use Case	Windows	Linux	Mac Intel	Mac PPC	Comment
1	Chandler startup with existing test repository
2	Create new event (double click)
3	Create new event (file menu)
4	Stamp an item
5	Triage dashboard
6	Create new collection
7	Jump calendar by one week (1st time)
8	Jump calendar by one week (2nd time)					Jump to the same week as in the previous test
9	Overlay calendar (1st time)					Overlay 2 of the calendars
10	Overlay calendar (2nd time)					Overlay the same 2 calendars as in the previous test
11	Publish collection
12	Subscribe to collection
13	Import a 3000 event calendar					Here's the test .ics calendar

Secondary Scenarios

id	Use Case	Windows	Linux	Mac Intel	Mac PPC	Comment
20	Switch view : dashboard -> calendar
21	Switch view : calendar -> dashboard
22	Scroll calendar
23	Scroll dashboard
24	Resize application in calendar view
25	Resize application in dashboard view

Pet Scenarios

id	Use Case	Windows	Linux	Mac Intel	Mac PPC	Comment
30	Drag event on Calendar Canvas
31	Switch between collection : calendar view
32	Switch between collection : dashboard view

• PerfDataSet002.chex: Perf Data Set (chex extension)

-- PhilippeBossut - 23 May 2007