r11 - 06 Nov 2007 - 12:58:50 - PhilippeBossutYou are here: OSAF >  Developers Web  >  ContributorNotes > PhilippeBossutNotes > DesktopReleaseProcess

Chandler Desktop Short Cycle Release Process

Interesting Links and Inspirations

Objective and Fundamentals

The Chandler Desktop team agreed on the following objectives and rationale for the new Release Process:
  • Shorter cycle / Release more often: we want to break from the long release cycle system we've been in, we want shorter releases in order to:
    • Get bugs found by users fixed faster (so users don't wait months for bug fixes)
    • Get new features requested by users released faster and more often
    • Manage expectations more smoothly (aka don't build huge expectations because releases are rare)
    • Allow experiments and iterations for features (see expectations here above)
    • Allow contributors to see their work picked up faster (more motivations to get things done)
    • Avoid long debug period slog
  • Be more reactive: we want to be able to post a quick urgent fix on an already released version anytime
  • Be more flexible: we want devs to be able to work on things (features, architecture, major rework) without stalling the project
  • Be truer to Open Source methods: we want the same process to work for everyone, being on OSAF staff or not, so new contributors can start something and be handled as anyone else (though they may not have commit privileges immediately but, essentially, there's no difference between contributors with commit privileges).

Fundamentals:

  • Adopt a Scheduled Releases Strategy: releases will be schedule driven, not feature driven
  • Work on stuff in Priority Order: features and work items are worked on in priority order

Decisions:

  • We'll do Monthly releases of 0.7.x
  • Big items will be worked on SVN branches
  • The SVN trunk is restricted (i.e. bug and code review required)
  • We are committed to make a concerted effort to support more auto tests (i.e. we'll dedicate devs to work on this problem)

Release Process

Release Schedule

The schedule of a release spans 5 weeks. The last week sees QA and development working in parallel so the effective release frequency is 4 weeks:
  • T0 = Start date: The start date for a new release is the date the previous release is branched off the trunk. The first such date was 10/03/07 when 0.7.1 was branched off and 0.7.2 started.
  • T0 + 2 weeks = Landing date: This is the last date development branches can be merged on the trunk. Big changes after that date will need to wait till the next cycle.
  • T0 + 4 weeks = Branching date: This is when the release branch is created off the trunk. This is the last date a bug fix can make it into the current release. This date is also the start date for the next release cycle (see "Start Date" here above).
  • T0 + 5 weeks = Release date: This is the date the release is officially posted for download.

How do we make decisions on what gets into a release?

The risk we want to mitigate is the one of working on things that are not important or missing on bugs that are blocking users. So, here's the process on "picking up things for a release":
  • Centralized list on Bugzilla: we don't want to miss things because they live on Bugzilla, Wiki pages, emails, Excel spreadsheets and the like. Everything we consider working on has a record in Bugzilla and, conversely, anything that is not tracked by a Bugzilla record doesn't exist.
  • Target_release is used as a first order sort: we use that field as a primary sorting mechanism:
    • ---: newly entered bugs are all assigned to "---". The Bug Council will triage those bugs during its weekly meeting. In case of emergency, anyone can hail a Bug Council member on IRC or email and ask for a bug to be processed speedily and be assigned to the next release.
    • Future: bugs that we shouldn't work on in the next few months are triaged to "Future".
    • 0.7.future: bugs that should get attention in the coming months are parked into "0.7.future" which acts as a waiting pattern for bugs and work items in the pipe.
    • 0.7.x: bugs that are actively worked on and scheduled to land within the current month (i.e. next release) are triaged to that milestone.
  • At the beginning of a cycle, some retargetting is happening:
    • Auto punt: all records assigned to the previous milestone (0.7.x) and not marked fixed by the Branching date are automatically punted to the next milestone (0.7.(x+1)).
    • PPD priority: at the beginning of a cycle, PPD and Bug Council decide what gets upgraded from "0.7.future" to "0.7.x" according to developers' swag, user feedback, criticity of the bug, etc...
    • Dev priority: developers can weigh in and get bugs or features moved to "0.7.x" based on their knowledge of the architecture and the areas they are reworking on. As a rule, any "0.7.future" bug is fair game and can be promoted to "0.7.x" by developers without Bug Council supervision anytime. For bugs marked "Future", developers should engage the Bug Council and the community at large on email lists or IRC to get the work item retargetted.
  • During the cycle, Bug Council will triage newly logged bugs to "0.7.future" or "Future". No new bug will be added to the current "0.7.x" release unless the bug is a blocker (crash, data loss), a fix for a newly added feature (from a branch that landed before the Landing date) or blocking user adoption in a big way.

Development Processes and Policies

SVN

We adopted what was described as "s3" in Grant's email. Risks we want to mitigate with that process:
  • We don't want to destabilize the code so much that debugging takes an amount of time we can't project
  • We don't want to make the process so complex that nothing moves...
Trunk
The SVN trunk is where the code of the next shippable 0.7.x version lives. The process goal is to ensure the stability of that code as much as possible while allowing developers to patch small bugs easily and swiftly:
  • The trunk is the most recent version of the upcoming official release.
  • The trunk needs to be evergreen so commits that break the build are rolled back automatically.
  • The trunk is open for bug fixes and small incremental improvements (small clean up and/or refactoring), big changes must happen on development branches (see Development Branches here under).
  • The trunk is restricted, i.e. a logged, triaged bug and pre-commit code review are required on commits to the trunk.
  • Merge of Development Branches to the trunk can be done only in the first 2 weeks of a release cycle, i.e. before the Landing Date.

Development Branches
Risky and/or big development jobs (being bugs, features or refactoring) are done on SVN branches. The process goal is to track and consolidate branches before merging them into the trunk:
  • New features and big changes are developped on SVN branches
  • Branches are created by developers on an as needed basis
  • Branches are created "per project" meaning that several developers can collaborate on a branch (IOW, we don't follow a "per dev" branch method common among other projects)
  • Each branch has a corresponding "branch_node" superbug record in Bugzilla so everyone can easily track which branches are under work at any time (see Bugzilla here under)
  • Branches are named using a "human readable name" and a reference to the Bugzilla branch_node record, i.e. name_bugnumber (e.g. "ItemLabelling_15678")
  • Branches are merged to the trunk (i.e. in the current trunk of the 0.7.x release) only when fully tested (see build and tests)
  • Branch merges are done only during the first 2 weeks of any release cycle (before the Landing Date) so to give time to backtrack if the feature is found too flacky to make it in the release
  • Branches are built and tested on TBoxes on demand. A branch build must be done at least once before merge.

Release Branches
Release branches are created automatically off the trunk on the Branching date, one week before release. As a matter of policy Release Branches are patched only rarely to fix blockers discovered during the acceptance test week:
  • Release branches are created on the "Branching Date", one week before the "Release Date". This week is used by QA to qualify the release (run acceptance tests on the 4 platforms)
  • Only blockers are fixed on release branches, the Bug Council decides on blockers, devs can nominate blockers directly in Bugzilla or contacting a Bug Council member (IRC, email, whatever...)

Bugzilla

We already talked about Bugzilla use while discussing the Release Process above so we're just listing here under the processes in use when modifying records in Bugzilla:
  • We create Bugzilla records for any item we work on (bug, refactoring, feature, etc...)
  • New bugs are always targetted to "---", they're triaged weekly by the Bug Council and can be triaged in faster just chatting Bug Council members on IRC or email
  • Each Development Branch has a "super Bug" associated with it that has the keyword "branch_node" set
  • Bugs that are being fixed on a branch should be marked blocking their respective "branch_node" bug and their target milestone should set with the same value as their branch_node superbug
  • Bugs that are fixed on a development branch shouldn't be marked fixed but simply marked with the "havefix" keyword
  • "branch_node" superbugs themselves are marked fixed only when the branch is merged into the trunk along with all blocker bugs
  • Only bugs marked "0.7.x" can be marked fixed into the trunk
  • Anyone (dev, QA, PPD) can promote a bug from "0.7.future" to "0.7.x" without formal supervision (Bug Council members are copied on changes anyway)
  • "Future" bugs need discussion on the dev or design list before being promoted to "0.7.future" or "0.7.x"
  • Once a release branch has been created, the Release Manager creates the next target release in Bugzilla ("0.7.(x+1)") and reassign the old non fixed bugs (open "0.7.x") to that new target.
  • Once the new target release has been created, the Bug Council reload the new target release with bugs picked in "0.7.future", taking into account the work load per dev and the swag

Build and TBox

The main innovation in the Build/TBox process is that we allow development branches to be built, tested and released. This is so to mitigate the risk of destabilizing the trunk while allowing thorough and early testing of a new feature:
  • Build and TBox continue to run on trunk automatically
  • On demand, development branches can be built on the supported platforms and unit and functional tests ran. Report is sent to the branch committers.
  • Also on demand, development releases can be created on branches so that the whole process can be tested and the community can download and test the new feature outside of the regular release process. The naming of dev releases is: 0.7.x.dev-rsvnnumber-branchnameYYYYMMDD
  • Development branches must be built and tested by QA at least once before merge
  • Check points are built off the trunk on a weekly basis. The naming of checkpoints is: 0.7.x.dev-rsvnnumber-checkpointYYYYMMDD

Communications

While we are building a community of users and developers, it's important to maintain the communication channels open and alive. We'll follow these guidelines:
  • The creation of branches for subproject is advertized on the dev mailing list with a reference to the Bugzilla branch_node, a short rationale and call for help (if needed)
  • Development branch readiness for testing and review needs to be advertized on the dev mailing list as well
  • Developers and QA use the #chandler IRC channel to communicate and sync up in real time
  • Bug Councils in particular happen weekly (Tuesday 4pm) on IRC for all to see and participate
  • The Desktop Release Teammeet weekly (Thursday 4pm)

Caveats / Up in the air

  • Should we do a daily sync up on IRC in the last few weeks, say after the Landing date, before shipping any 0.7.x? (like in a scram process)
  • What if we end up with too many branches? Each could diverge in substantial ways making sequential merges difficult. A way to mitigate that is to make sure that branches have as little code overlap as possible (difficult to guarantee though for architectural and refactoring projects).
  • Development branches might clash with Cosmo or the Repository making the production and release of developers releases risky. Those risks must be carefully considered and communicated when creating those releases.

-- PhilippeBossut - 15 Sep 2007

Edit | WYSIWYG | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r11 < r10 < r9 < r8 < r7 | More topic actions
 
Open Source Applications Foundation
Except where otherwise noted, this site and its content are licensed by OSAF under an Creative Commons License, Attribution Only 3.0.
See list of page contributors for attributions.