How to improve and facilitate prioritization of bug fixes?

Question

In a long-term software project with frequent releases, end-users regularly report bugs and defects in the product. When the reported bug is highly critical, the fix is generally developed and released right away. Otherwise the bug is recorded into a bug-tracking system, which is basically a list. One of the tasks of the product owner is to prioritize this list and decide what fixes to include in the next release.

The difficulty of prioritizing defects will increase along with the number of reported defects. Specifically, it is difficult to:

• Estimate implications of each bug (including risk, technical complexity, etc.)
• See the whole picture : How stable is the product? Did the quality drop with the last release? etc.
• Identify duplicate bugs

What process or tool can the product owner (and the team) implement to facilitate (and potentially improve) prioritization of bug fixes? What are the advantages/disadvantages of these processes or tools?

Answer 1

With my team we recently adopted a new bug-tracking tool based on a criticality matrix. This is a physical board representing a 4x4 matrix on which we stick notes according to two criteria:

• Columns represent the intrinsic severity of the bug. The more a bug is on the left, the more it is critical. Each column has its own explicit policies but, for example, if a bug can cause the loss of users' data it will be in the leftmost column, whereas a very rare "cosmetic" bug will on the rightmost column. Deciding in what column to put a bug is the responsibility of the product owner.
• Rows represent the team feeling about the bug. The more a bug is on the top, the worse is the team feeling about it. This evaluation is quite subjective and includes things like risk, technical complexity, time to develop a fix, etc. Choosing in what row to put a bug is most of the time the result of a 30 second discussion. The topmost row is dedicated to bugs that need an investigation before we can decide, typically when we completely ignore the cause of the defect.

As we always have quite the same number of bugs, we needed a way to measure the overall stability of the product, which depends on bugs criticality. So we affected an arbitrary score for each row and column, allowing us to calculate a stability score.

Advantages

• Visualization makes prioritization easier
• Everybody understands why a bug has been chosen instead of another, as the choice is often obvious
• Easy to implement when everybody work in the same place (maybe easier in a Kanban team like ours)

Disadvantages

• Probably impossible to handle a big number of bugs - say more than 200 - with a physical board

This blog article might give you more details.

Answer 2

TL;DR

The problem isn't with your bug-tracking tool chain. The underlying problem is a lack of rigor in defining bugs in your business context, and possibly a lack of clarity in the business objectives that your organization is trying to achieve.

The solution is to define your terms unambiguously, clarify your objectives, and then build processes (including bug tracking and change management) around those objectives in a sustainable way for your particular business.

What Bugs Are Not

Your question mashes a bunch of things together. You said that it is difficult to:

• Estimate implications of each bug (including risk, technical complexity, etc.)
• See the whole picture : How stable is the product? Did the quality drop with the last release? etc.
• Identify duplicate bugs

You're confusing bugs with product stability, business risk, and work effort. While all of these things may be associated with a bug, they aren't synonyms. So, let's start with a workable definition.

Defining a Bug

Here's my personal definition of what a bug is:

A bug is an aspect of a system that performs improperly (e.g. 2 + 2 = 5) or fails to meet requirements or customer expectations (e.g. the specification is for 10ms response times, but printing "Hello, world!" takes 17.5 minutes).

Your project may have a different definition. That's fine; the important thing is for everyone to agree on what it means when you call something a bug.

Bugs Need a Point of View

A bug needs a point of view. Generally, a bug that impacts no one isn't a problem, so it's important that a bug have a context and a point of view. A bug that electrocutes end-users is probably a business-critical bug that needs to be fixed right away, while bugs that are not customer-facing will probably have a different priority for the business.

If you don't understand who the bug impacts or why it matters to them, then you can't identify business risks or quantify the value of fixing (or not fixing) a particular bug.

Customers and Managers Don't Get to Estimate Complexity

While it may be legitimate to allow the bug reporter to provide information on the severity of the bug's impact on a given person or process, that's not the same thing as estimating business risk or work effort. The best way to make sure you estimate something poorly is to have the wrong people provide the estimates. As a result, initial bug reports shouldn't contain estimates of risk or complexity; it's the team's job to evaluate those issues.

In Scrum, the Product Owner should determine what the business risk and criticality is of the bug. She may or may not consult with the rest of the team if the risks are wide-reaching or non-obvious, but ultimately prioritizing things is her job.

Likewise, only the development team gets to estimate the complexity of the bug. After all, they wrote the software--and presumably the unit tests that can at least reproduce this bug--so they are the best people to estimate how difficult and/or invasive a given change is likely to be.

Don't Ignore Duplicate Bugs

Cross-referencing bugs is fine. However, automatically de-duplicating bugs is almost impossible. Either you end up with a bug reporting system that's so limited that only pre-defined bugs can be entered, or the free-form nature of the reports makes finding exact duplicates impossible.

Every bug needs some human triage. If bugs are related, it's okay to mark them as similar or exact duplicates of some other bug, but the fact that you have 973 "duplicate" bugs about your Super-Duper Widget is probably a useful metric in its own right. Why would you want to discard such valuable data?

Bugs are Not Intrinsically Code-Quality Metrics

A bug report does not intrinsically tell you anything about either code quality or product stability. A code base riddled with easily-preventable bugs is obviously a red flag for your process, as is a high volume of bugs that crash the application or its host system.

However, some bugs are just a result of emergent system behavior, changes in user expectations, poorly-chosen metaphors and paradigms, or simply unforeseen circumstances. These things are definitely bugs, but they don't mean your code base is bad, your programmers are lazy, or that your customers are idiots. They are, at heart, just specifications for change requirements or new user stories.

If you want to measure code quality, there are other ways to do that. However, I'd argue that end-user satisfaction and code base sustainability are the metrics that matter most. Measure those two things properly, and life will be easier for everyone.

Answer 3

Limiting my answer strictly to the PM implications (there are others).

Fundamentally a bug is a change proposal; part of the PM's job is to analyze the implications of a change proposal. I think that one method would be to consult with the change management stakeholders (you've mentioned the development team, but I would include customer representatives, et. al.) and assess what the impact of the bug is on the software, including estimates of the resources required to fix it. (I'd also search for questions that reference technical-debt. Both Risk Management and Change Management sections of any PM reference contain methods to solve these problems.

You seem to indicate (either your grammar or my lack of caffeine is confusing me), that you're having trouble getting technical advice from your team; if that is so, then I'd table this problem and work full time on improving team communications. The other two difficulties you cite can probably be better addressed in a software specific forum (I would suggest you consult CVE/CWE, or Software Engineering Institute models, and adapt those.)

Assuming that the problem isn't team communications, it is about how to structure the conversation and quantify bugs, then I'd suggest either risk management or a variation of planning poker. Or you could use the Delphi technique. Or you could play price is right "Which of our existing bugs is closest in severity to this bug, but not more severe?" (If it were me, I'd suggest that the dev's spend no more than 1 minute on the decision.). Or you do risk management, and estimate the probability that this bug will ever be triggered, and the potential consequences of the bug. Possibly calculate an expected value for the bug.

The point is that in addition to answers you get from software experts, you can use either change management techniques or risk management techniques to do this.

Update:

I still think that the software development world contains the core of the answer, and if I were in the OP's shoes, that's where I'd start my research. I'd also consult QA: they should have internal procedures that would help. I'm going to ignore that and answer with my PM hat.

1. I think the key is to establish a process that is sufficient for your team - one that will produce an answer you can live with, but doesn't consume more time and resources that it produces value. Kind of obvious, but I don't want to miss the obvious. Ultimately the accuracy of the process is probably less important than (a) the consistency and (b) the amount of time it takes devs away from dev work.

2. I'd probably start by emphasizing lightweight. Ask each dev to spend no more than 1 minute examining each bug, and predict how many hours of coding it would take to fix the bug. (Emphasis on quick SWAG). I'd drop the high and low, average the remainng estimates, and use the total as my total technical debt. Personally, I'd probably also track the accuracy of estimates and start applying a weighting factor as I learned which of the devs are most skilled at estimates. I'd also incentivize correctness - I'd buy lunch for whoever is closest to the actual number. Of course the drawback is that I've omitted the consequence of the defect, but I've got a pretty good idea of what the impact is on schedule.

3. For impact/consequence, I would consult the customer. (First data mining the bug report database. Look for clusters that might be identical; then find a dev who needs a coffee break and ask for help in identifying duplicates.) Based on bug reports, I can estimate how frequently this defect is likely to be triggered. (I'm restricted to my PM hat, but if I were wearing my software hat, I'd also look at code profiling to see how often the code is triggered). I'd also sit down with the product owner/business process owner and discuss whether the defect affects something that is core to the competitiveness of the product, and weight things that way. Then I'd move to a spreadsheet and try to build a model - if the sales are based on X sales at Y dollars, and the bug will affect a/X customers, then I'll treat the value of the bug as f(X,Y,a). This is very rough, but it involves very little of the stakeholder's time.

4. If you combine the above, you've got risk management. I think you could probably adapt one of the more mature risk management methodologies (FAIR is the sexy one right now, and has the advantage that it is relatively comprehensive. Start there and start stripping things out that don't contribute enough value.

Ultimately though as I've said elsewhere, this is a core job function for the PM. This is where we earn our big bucks, and if it gets too simple, then they won't need us.

Answer 4

One of the main problems with prioritizing bugs as they're reported to you by users is you're missing a lot of information, namely, how widespread a bug is and what its impact is on your software. When you rely on bug reports from users, you are at the whims of whatever seems to be the most pressing problems based on whoever has the loudest voice.

Rather than prioritizing bugs this way, (which, you're right, does make it a challenging task), you might want to try proactively detecting bugs with an application error monitoring tool that will give you added context on every error. This will make prioritizing bugs more straightforward.

With an understanding of all the bugs that exist in your application, things that are helpful to consider are:

• How often is this error occurring?
• How many users are impacted by this bug?
• Have I tried to fix this bug in the past, and I'm now seeing a regression?
• Is this bug affecting my most important/highest paying customers?
• Is this bug happening in an important part of my application, like a signup or checkout page?

This technique isn't without its faults. Proactively detecting bugs means more noise, and alert fatigue is something to be wary of. There are things that can help you make sense of the noise:

• Grouping like errors together
• Intelligent alerting so you only get notified when there are new errors, error rate spikes, or error regressions
• Silencing or snoozing errors that you don't need to fix immediately
• Consolidating errors in one place

Further reading in A guide for prioritizing bug fixes