Commits and Notifications

Participants: John, Katie, Grant, Andi, Jeffrey, Philippe, Bryan, Brian, Morgen

Agenda

• Commits and Notification: large commits create tons of notifications and lock the UI. John proposed to block notifications for a while. We'll discuss the issues of this ideas and alternative strategies to avoid notification storms.
• Merges and Reindexing on Refresh: if time allows, we'll discuss the a sibling issue of merges and reindexing on refresh that are the cause of slowness.

Notes

Commits and Notifications

• John's idea: blocking notification for a big reload and rebuild the indexes at the end
• Andi: will block later...
• Andi: we're not rebuilding indexes, just updating them
• Andi: Changes since Tuesday: currently you can't turn off refresh notification 'cause indexes relies on them to sync. Andi thinks we can now track indexes and sync them instead of relying on notification for this to happen. This will avoid indexes becoming out of wack (see Bug#7324 on that) and should be tons faster. So index sync won't rely on refresh notification anymore.
• John: should we rebuild from scratch? Andi: that doesn't scale, changes on 3 items is way faster than rebuilding a 20,000 item index...
• Idea: May be the UI should unsubscribe for some notification? If done, some notification won't simply happen (can be skipped)
• Andi: Could send the number of notifications to come. Useful? Not that much 'cause they might be non redundant
• Idea: Register for some type of changed
• Idea: get a threshold of notification count above which we'd just get a single "wxSynchronize everything" notification. (As long as we’re dealing with synchronous notifications, the repository knows the item count when it’s about to process notifications, so the UI could check that).
• Grant: when are notifications sent? Where's the expensive part? Andi: history iteration is the expensive part
• John: we could mark block as needed to be redrawn. Or tell the repository to stop sending notifications once all blocks are dirty.
• The upside of all this is that we could make Chandler more responsive when large numbers of items are being imported from a background thread/view. Clearly, this would be needed, say, in a scenario where email is involved. Katie: This could also be dealt with via a progress dialog.

Merges and Reindexing on Refresh

• Item merge: Changes are rerun (merged) on freshly reloaded items
• Index merge
• The UI should do small commits so to do small merges
• Issue with generated occurences: create a bunch of new items (instances) to merge
• Are we creating too many sub-indexes? In the sidebar for instance or in the dashboard ... Andi noted that with master index pre-creation disabled (to reproduce a different bug), he quickly saw 29 indexes created when clicking around in the dashboard.
• Idea: may be create only one sub index on demand (when sorting by said column) instead of having all sub indexes? Unclear it'd save much time though (could be...).
• See Bug#8319 for more on this
• Andi: There are cases where the UI view makes changes that don't get committed. This means there’s more merging done. The main case is the creation of occurrences when you click around in the calendar.
• There was some speculation that, if you had a recurring event that started in the past, we would create all the intervening occurrences when displaying the calendar. Post-meeting, Jeffrey and Grant determined that this was not the case.

Other:

• Brian Kirsch noted that, after importing a bunch of mail into Chandler, memory usage was around 300MB, and didn’t drop down until quit and restart. It was speculated that this was a result of python not returning memory to the OS, or possibly some issue with PyLucene (the gcj garbage collector) and/or indexing.

Next Steps

• Andi to commit indexing changes (i.e. not being dependent on notifications)
• Block dirtying: John to write up a bug for Bryan to investigate.
• Someone (Grant?) to do some timings, e.g. of how long a complete sync of the UI takes. Possibly look at non-persistent dashboard indexes.

-- PhilippeBossut - 17 May 2007