Tracking Tasks and Bugs with Bugzilla

Introduction

Right from the beginning of the Chandler 0.6 release cycle, the App team started to accumulate a large amount of small bugs and tasks. We rapidly felt that it was becoming increassingly difficult to make sense of the various places where we tracked stuff and that we needed a unique tracking place so that we can keep our sanity (which is a great goal all by itself) and monitor the status of the work in real time.

Bugzilla being already installed, used for bugs and familiar to all, it seems like a good idea to use it to track everything, including tasks that stem from specs. This however will require some modification to the Bugzilla fields and agreement on some bug and task cycle conventions.

The first thing in order is to make sure we use the existing bug fields correctly (rather than relying on massive keyword convention for search and sort) and use common conventions on bug logging so that automatic report and tracking queries can be written.

The second is to make sure we can map any task required in 0.6 to a record in Bugzilla. This will require some work in Bugzilla to update some fields (component list, keywords, etc...).

The third and last thing is, of course, to add the task record to Bugzilla and create the relevant tracking queries.

The Bugzilla Fields

Most of the fields are self explanatory (e.g. "OS" or "Status") and we assume in this document that all Chandler developers are familiar with Bugzilla normal flow of operation.

If you need more detailed information on each field in Bugzilla, please check our Bug Writing Guidelines.

We will concentrate here under on the fields that are used for tasks and bugs tracking for 0.6 and discuss the conventions we are using.

Since we are going to use Bugzilla more extensively and track more things, we need new values for some fields. Those new attributes are identified by this style in the document.

Components

Components should map specs (within a single release cycle) as much as possible. Since new bugs logged against a component have a default owner, this should make the spec/bugs/tasks ownership relatively clear (at least for 0.6).

There are already components existing for identified 0.6 specs (like Detail View for instance) but we need to create some additional ones to cover all 0.6.

Also, with a proliferation of components, it's convenient to group them in a rational way per functional area rather than alphabetical so that bug reporters (especially people outside the core groups) can find their way around easily. Bugzilla does not provide a way to organize components in a hierarchy. The Mozilla convention is to use a "Main : subcomponent" name so to create grouping throught alphabetical order. Chandler is already using this convention for Repository and Documentation. We propose to use the same convention and, in the process, rename some old components and give some order to the list.

Here's a proposal:

New Component name	Old Component name	0.6 Spec	Default Owner
Calendar	Calendar	Calendar	Alec
Calendar : Minicalendar	n/a	Calendar	Jed
Calendar : Recurrence	n/a	Recurrence	Alec
Chrome	Chrome	Visual Guidelines	Jed
CPIA	CPIA	CPIA documentation	John
CPIA Script	n/a	CPIA Script	Donn
Detail View	Detail View	Detail View	Bryan
Drag and Drop	n/a		Donn
Sidebar	n/a	Sidebar	John
Summary Table View	Summary Table View		John
Toolbar	n/a		Philippe
Trash and Deletion	n/a	Trash and Deletion	Alec
wxPython	wxPython		David

Notes:

• This list omits the components which default owner is not part of the App team
• This list mentions only the components that are planned to be improved in 0.6 by the App team
• This list has been reviewed in meeting with the App team
• New proposed components are displayed using this style

Priority

The Priority field is used by the dev team and project management to sort which bugs must be worked on first. "P1" is the highest priority, "P5" the lowest.

The Priority is a scheduling field and should not be confused with Severity. When reporting a bug, you should not be concerned by the Priority. For bugs with a "New" Status, Priority is not meaningful yet.

In 0.6, we are using the full range of priority values so that to make the scheduling of tasks and bugs very clear to all.

Severity

The Severity is used by QA and bug reporters to communicate how bad the bug feels. It must be set when logging a bug.

While this field has not been used consistently in the past, we are using it in 0.6 so that we take full advantage of that dimension to sort and track issues (especially "New" issues).

Note that tasks (see under) as opposed to bugs always have Severity set to "enhancement" but, of course, not all enhancements are tasks...

Keywords

Keywords can be (and have been) easily abused. They should not be used for things that can be set on/off (like "0.4Demo", use Flags for that), neither should they be used to duplicate other fields (like "crash", Severity should be used here).

Keywords should reflect the semantic nature of the bug/task so they typically describe properties that do not change (unless of course the understanding of the bug change...). We use keywords so that we can retrieve and sort bugs in categories that are meaningful to us (devs or QA).

We are going to use Keywords for 2 purposes in 0.6: task tracking and bug/task typology.

task

Since we are going to use Bugzilla to track dev work item and not only bugs, we need a way to identify those unambiguously.

For this we will be using a new keyword, task.

This keyword identifies records that are not bugs but development tasks, work items needed to be completed to implement a spec. As a rule of thumb, bugs should not be filled on non closed tasks (see The Task Cycle here under).

Type

Keywords are also currently used to provide a typology of bugs. This is useful when trying to sort bugs on a dimension that is orthogonal to components but useful to devs and engineering management.

Here's a proposed enhanced list for 0.6:

• behavior : denotes bugs/tasks that are dealing with the behavior of the application (e.g. Drag and Drop, delete, double click, ...)
• visual : denotes cosmetic issues (e.g. icon issue, typo in text, ...)
• rendering : denotes display issues (e.g. something does not display when it should, display is broken, ...)
• perf : performance issue
• infrastructure : denotes issues dealing with general non visual aspects of the code (e.g. CPIA, wxWidgets, ...)
• devitem : denotes issues relevant only to developers (e.g. asserts, code structure, ...)

Note that setting up Keywords is optional. Most of the time, they will be set by developers after reviewing and exploring the issue so we can track some categories of bugs. Also Keywords are non exclusive so one bug can belong to several categories.

Status Whiteboard

Eventually, we plan to add a new feature in our Bugzilla bug recording page so that we can evaluate bugs (SWAG) and keep their estimate consistent. In the meantime though, we are following the Mozilla practice of putting the SWAG in the Whiteboard status field.

We are using SWAG for both tasks and bugs. The 5 values we use are:

• [SWAG : Trivial] : bug that takes a few hours to fix
• [SWAG : Small] : 1 day to fix or implement
• [SWAG : Medium] : 2 to 3 days to fix or implement
• [SWAG : Big] : 1 week to fix or implement
• [SWAG : Huge] : 2 weeks or more to fix or implement (those are good candidates for spec and further decomposition)

We also use this field to indicate (as part of the SWAG) the level of confidence we have in our estimate:

• [Confidence : High] : perfect comprehension of the work at hand, SWAG is between 80 and 100% accurate
• [Confidence : Fair] : good handle on the bug or task (typical), SWAG is between 50 and 80% reliable
• [Confidence : Low] : poor handle on the bug or task, SWAG is only 20 to 50% reliable
• [Confidence : Nil] : risk item, no or very poor visibility of the work to perform, SWAG (if any) unreliable

Optionally, devs can also indicate the completion date (aka ETA for Estimated Time of Arrival) using a similar pattern:

• [ETA : mm/dd/yy] : date of completion in month/day/year format

Note that using a "[attribute : value]" syntax is also a Mozilla practice.

Estimates

The estimate field is currently in hours and is somewhat finicky to use and maintain. We have decided not to use it right at that stage.

Still, it's nice to leverage its existence and possibilities to track progress. In the future, we plan to add a "SWAG" drop down in the Bugzilla bug record page that will run a small JavaScript logic to automatically update this field.

Dependent and blocking bugs

Bugzilla allows to track bugs against each other using dependencies upward and downward. It's a facility that can easily be abused (to create a convenient dev oriented ordered list of bugs for instance). We should only link bugs that are really linked to each other through code (i.e. bug or task A cannot proceed to the next level before bug or task B is completed).

In creating tasks though, it allows us to create amorphous "big" tasks (also known as "Superbugs") and cut this task later by creating a list of subtasks that will be listed in the dependent list of the Superbug.

Note that if you do something like that, make sure you nuke or otherwise reduce the SWAG of the Superbug when SWAGing the subtasks. Failure to do so may artificially inflate the SWAG count used by automatic reporting tools.

Target milestone

Like the Priority field, the Target milestone field is a scheduling tool used by the dev team and management (bug council typically). It is used to spread the work load during a major release through intermediate milestones.

OSAF is using the following convention for release and milestone numbering:

• a major release is coded "0.n Release"
• the intermediate milestones leading to this release are coded "0.(n-1).x"

So when working on release "0.6" for instance, most of the bugs will have a "0.5.x" target milestone. The bugs and tasks still marked "0.6 Release" can be considered as part of the release plan but not scheduled yet. Bugs punted to a next release need to be set to "Future".

When logging a new bug or task, you should use the next appropriate milestone (if you know when the feature is scheduled) or the next major release (i.e. "0.6 Release" in our example). The bug or task will be triaged, SWAGed and prioritized by dev later. The dev and dev management will then decide in which milestone the fix will land, spreading the bugs and tasks along the several intermediate milestones set for the current release (e.g. "0.5.02", "0.5.03", etc...) or "Future" if the issue cannot be addressed in the planned release.

For a list of the current and future milestones, check the Chandler Milestones Schedule

Also, for more information on release management, check OSAF Release Management with Bugzilla.

The Bug Cycle

Logging a Bug

We won't rehash the general rules already available in the Bug Writing Guidelines. Just make sure that you fill the following fields using the convention discussed in this document so that we can search them and report on them efficiently. Namely:

• Use the correct Component. Use the here above mentioned 0.6 spec list matching matrix if you have a doubt.
• Leave the Priority alone... Use P3 as a default.
• Set the Severity carefully! This is going to be used in reports and tracking.
• Set the dependent and blocking bugs if any
• Set the target milestone to the next major release: add in comment if you feel that the bug must be moved to an earlier milestone (in particular if it's a severe bug)
• You don't have to set Keywords unless you really know what you're doing...
• Leave the Whiteboard status blank

That's it. When reviewing their New bugs (moving bugs from New to Assigned status), devs will:

• Set the Priority according to the schedule, project tenets and dependencies.
• Use the Whiteboard status to SWAG the bug.
• Set the target milestone to the relevant milestone according to the schedule.
• Optionally, set Keywords.

Bug Life cycle

None of the convention discussed in this document modifies the Bugzilla bug life cycle we've been using so far. For more details on a Bug Life Cycle in Bugzilla, please check out Chapter 6.4 and figure 6.1 of the Bugzilla Guide.

The Task Cycle

Logging a Task

Note that only devs can log tasks. We use tasks to track and manage our own work load so, please, if you're not a dev, do not log tasks. If you want to make a suggestion for improvement, simply log a regular bug and use "enhancement" for the Severity field.

Apart from this, logging a task is not very different from logging a bug. Pay attention to the following fields so that the report and tracking tools will work correctly:

• Set up the Component field to the relevant value.
• Set Severity to "enhancement" (unless you have a very good reason to do otherwise).
• Set Priority according to the schedule, project tenets and dependencies (rule of thumb: higher priority bugs must be worked on before lower priority ones). Use the full spectrum of priority values.
• Add task to the Keyword list. Failure to do that will make the reporting tools miss your task and reporting it as a bug.
• Set the dependent and blocking bugs, especially if you're defining a task as a subtask of a larger one
• Set the target milestone to the next major release or any other intermediate milestone as appropriate
• Optionally, add other Keywords to characterize the task.
• Use the Whiteboard status to SWAG the task.

Life cycle

Since tasks are opened by devs for features not yet implemented, their life cycle is slightly different from the normal bug life cycle:

1. Development subcycle : during this subcycle :
   • The Bugzilla record is simply kept in the Assigned status
   • Devs modify the other fields as they see fit (Priority can change as well as SWAG and even Target milestone)
   • Comments are used by the team (dev, QA and designers) to communicate about the task so to track discussions and decisions
   • Bugs are not logged against the task. This is a rule of thumb of course. If a serious issue is created by check ins related to the task (in particular, crashes in other features or other regressions), bugs can be filled.
2. Closing subcycle : once the dev and QA agree that the task is complete, the record joins the normal Bugzilla life cycle :
   • The dev changes the status to Fixed or other adequate status so to move the record to a Resolved state.
   • The QA moves the bug to Verified or Reopen after testing as appropriate.

Note that it is advisable to close the task (which is generally all encompassing and large) and open new bugs to track issues following implementation rather than leaving the task open forever and tracking micro changes from within. Task can be closed once the code associated with the record is complete which does not mean bug free.

Using this convention, we will be able to track code completion separately from shipping status.

Tracking to Subversion Changelist

Possible? TBD when we move to Subversion...

Reports and Charts

TBD. We will add a range of charts to the Apps team page and will be using them as a sort of Dashboard for the project.

Here's an example of such a report, courtesy of Alec:

• Bugs per dev in each component

Conventions and Idiosyncrasies used by the Chandler's App Team

OSAF has some restrictive use rules of Bugzilla's fields that we should all know.

Links

• Bug Writing Guidelines: advices from the Mozilla world.
• How to Write Useful Bug Reports: actually lists some Bugzilla field convention in use at OSAF
• Chandler QA Handbook: about the QA process in general
• OSAF Release Management with Bugzilla
• Bugzilla Guide: from the horse's mouth...

---

---

Philippe: We had a meeting on Thursday April 28th to review this doc. We took pretty much all the here under feedback into account. Check the notes of this meeting^? for more details. I also added comments here under for those who didn't attend in person (Heikki and Donn).

Heikki:

• Personally I tend to use Severity: Feature quite little. When you are starting a project, everything would seem like a feature, but I think that the core unimplemented features are in fact bugs. I think of the non-core features that are not yet implemented more like features. Typically I use a feature as a feature request - nice to have, but I can live without.
  • Philippe's answer: It's a matter of convention. We all agree that we need a way to tag work items differently from bugs and we can't do that with severity so, the question is: what's the severity value we use for work items (aka tasks)? Of the currently available values, enhancement (BTW, there's no feature value for severity) seems to be the most appropriate.
• I am not sure I completely understand the task flag. My gut instinct would be to not do this, or alternatively create a task keyword.
  • After some more thought, maybe you meant meta bugs (sometimes called superbugs). So for example we could have "Ship 0.6 Release" as meta bug, and all tasks and bugs that need to be completed/fixed before the release happens are marked as blocking that meta bug. I think this is a nice way to track bugs and non-bug tasks near a release, but trying to do it for a whole release cycle seems too much work (and too much bug spam). There is a slight difference between flagged bugs blocking a release and meta bugs.
  • Philippe's answer: Agree, using a flag is too much. We all settled for a task keywork
• Another common use of status whiteboard is [ETA date], which gives a concrete, actual date on which the bug is expected to be fixed.
  • Philippe's answer: Interesting, we may use that.
• I am not sure about the SWAGs... You don't define them, and it feels like there are too many. Also I would think that [SWAG Trivial] would usually be assigned to bugs where the severity is trivial.
  • Philippe's answer: [SWAG : xx] are loosely attached to the amount of time it takes to fix. We use Trivial for those bugs that takes a couple of hours to fix (resource change for instance). They're not necessarily non severe (e.g. exception being missed). I'll detail the meaning of our 5 values in the next iteration
• I am generally against adding too many keywords, because having too many just makes the system harder to use, and you never remember to set all of them anyway. I like the crash and dataloss keywords because both would typically be marked as Severity: Critical, but often we are more inclined to fix one type of bug over the other.
  • Philippe's answer: Fair enough. The idea is to caracterise the bug from a code standpoint, something that the component does not capture (e.g. performance)
• I like to be strict about adding new components (because removing them later is hard). I'd suggest trying to live with as few existing components as possible, and seeing how many bugs we are getting - if there is a barrage of bugs that warrans a new component. For example, I would be inclined to not create MiniCalendar component or Recurrence component, and file these bugs under Calendar. Only when we get too many MiniCalendar or Recurrence bugs in a short time should we create new components. Likewise components should start out as pretty general, and once we see what kinds of bugs are being filed can we make more edugated decisions about creating new bugs. Finally there is the factor of how easy it is to figure out which component to choose when you file a bug? Typical end users (and even developers who see bugs in areas they are not familiar with) will find it easiest to file a bug based on the part of screen where the bug happened: for example calendar, sidebar, email.
  • If there really is a need for recurrence, but it only happens with something users do from the calendar, an option would be to create component Calendar: Recurrence. Bugzilla does not have too many levels of hierarchy, and this colon in the component name is a convention to create one extra level of hierarchy.
  • Philippe's answer: we discussed that list in an App meeting with all the devs and came up with a shorter list. We will be using the component: convention. I'll update that list in the next iteration of the doc.

Alec's comments:

• I agree with heikki that the "task" flag might be better off as just a keyword. You can still query on it, and its just another aspect of the flags
• Recurrence and MiniCalendar are hard - Recurrence is actually a pretty big chunk of work relative to the rest of the calendar (going forward, it will probably account for more than half of the calendar work) MiniCalendar, maybe not so much.
• I don't see a serious need to distinguish between wxWidgets and wxPython personally, but david and jed may have more to say in that area. Frankly, I think we'd be better off creating a seperate product for wxWidgets if we need to track changes in the widgets code itself
• I think it might be important to talk about who owns what field. Typically
  • Engineering owns priority - to determine the priority of the work to be completed
  • QA owns severity to indicated how badly this bug/feature is impacting the product
  • Target Milestone is an agreement between Engineering and Product management, to help track when features will actually land
• The Keywords are good, but I think we need clear descriptions of them before we decide on a final list. Like, I'd like to know the difference between visual and rendering. It may be clearer once there is a description but right now they sound similar. I also have no idea of the difference between devtask and featuretask. I might suggest we nuke (or hide) a few of the existing keywords like 0.4Demo, etc, to address some of Heikki's concerns above.
• I think one important thing for everyone to realize is that not everyone is going to use every field. For example, I might put [SWAG Trivial] into a bug, and never use that again, but Phillipe may use that on a daily basis to help with bug metrics. I'm guessing QA will set the Severity field, and product management or Phillip/Lisa will track which bugs are labeled "Blocker" but an engineer might never even look at that until his manager says "Yo, get your act together, you've had a blocker on your plate for 2 days"

• Philippe's answer: Agree with almost everything and discussed during the App dev meeting. Thanks for the feedback!...

Bryan's comments

• I don't think we should track estimates separately from the SWAG values. I don't know how bugzilla would store the SWAG values, but we're already using a mapping between SWAG name and expected implementation time (Trivial = <1 day, Small = 1 day, Medium = 1-3 days, etc)... if Bugzilla could store the max day estimates as the value for this field, and map them to strings for the popup, it'd be easier to summarize remaining work based on SWAGs.
• Further, I don't like the idea of tracking "work remaining" on a bug - it'd be so inaccurate as to be useless. If we're tracking work items in Bugzilla, I'd rather create additional dependent work-item bugs.
• Also, I think Jeffery or Alec ought to own Recurrence, not me...

• Philippe's answer: Agree with almost everything and discussed during the App dev meeting. Thanks for the feedback!...

Jed's comments

• The Bugzilla Fields: I don't believe there is a "Platform" field anymore. It's broken into hardware and os. (just fyi)
• Components: Are there contacts bugs or feature requests? I don't feel like we're planning on doing any contacts work for 0.6, but perhaps I'm missing something. Same thing for Instant Messaging and wxSimpleCanvas.
• Severity: Right now I don't display severity in my bug queries because they're all normal. If something is labeled as critical it gets colored, but I'm essentially ignoring severity. Should I not be doing that?
• Bug Flag: I think you should be able to file bugs against non-closed tasks. I would rather have "extra" bugs than to lose any information.

• Philippe's answer: Agree with almost everything and discussed during the App dev meeting. Thanks for the feedback!...

Donn's Comments:

• Overall, I'm concerned that this may be too much process for us.
  • I think using Bugzilla is only appropriate for the larger tasks on our list - there are lots of small tasks, and routine tasks that should not be tracked with this level of detail.
  • I think the act of tracking needs to be lightweight, or we won't do it, or it will be done so infrequently that the information will always be out of date.
  • So, to that end I'm suggesting that we just try to capture the essence of the major tasks using Bugzilla, and not sweat the details. At least not right away. Lets start using it for a while, and then we'll find out which details are important, and which can be ignored.
• Given a limited initial use, I think this idea has many positives, and I look forward to seeing how it works out for us.

• Philippe's answer: I certainly share the concern of lightweight process and making it lightweight is the goal. The risk of having nothing is that we never have a clear picture of what's done and what's left to do (beyond a vaguely collectively shared hunch...). An alternative is making regular "checkpoints" but getting a clear picture for those checkpoints might precisely take more time than having a simple tracking system... Frankly, using Bugzilla to describe tasks and marking stuff as "fixed" when done is as easy as writting down a list of things to do on a piece of paper and putting a check in comment when submitting in CVS. Considering that each dev has roughly 30 bugs and 50 tasks, that's not a lot of items to track. I'd be concerned if the number of items per dev would grow beyond 100. If we get there, we should consider scaling back or use something else to report/track progress.