Best Metrics to Show the Value of Fixing Bugs

Question

My software development group has spent a significant amount of time recently on fixing existing bugs instead of working on new development.

Using our bug tracking software, I can get data on how many total bugs were fixed by time period and which areas of the software had the most bugs reported.

What is the most useful information I can extrapolate from these results? I know there is a lot of material available about performance metrics, but I am trying to get ideas on which would be most helpful for this situation.

Essentially, I want to be able to demonstrate the effectiveness of the recent time that we've spent dedicated to bug fixes. For me, it's not enough to just say that we were able to fix more bugs, I want to be able to show a dramatic overall improvement in the software. For example, would it be best to measure the decreasing rate of customer complaints? If not, what is the best application of data related to software bug fixes?

Answer 1

If I understand correctly you want to highlight the benefits of this bug fixing period over the usual mix of development and bug fixing. It is important to have a good baseline of metrics so that now you can show improvement. As you have already obtained permission for this period you must have used the existing metrics or maybe you are asking because you didn't have good metrics but the situation showed a sense of urgency that required this break on new development. Under any of this scenarios, it can be interesting to measure (show improvement in):

• decreased the number of bugs found in future releases as a consequence of the team working on a more stable code base, and (if this was the case) time spent on refactoring and archictecture improvements.

• decreased the ratio of bug reopening ( a bug that is supposed to be fixed but is reported again) versus the usual.

• performance and usability improvements are also a good point to put up front if there was some effort on this direction.

• bugs found by the team (during this phase, in which probably you have also done a more throughout testing ot the solution) compared to bugs found by customers.

A reduction of users complaints is definitely a good point in your favour but, as has been mentioned, you need to make sure on the real cause of this reduction and measure that your software has increased or maintained usage ratios.

Answer 2

Essentially, I want to be able to demonstrate the effectiveness of the recent time that we've spent dedicated to bug fixes.

It's obviously good news that you've fixed all these bugs, but as a manager/senior stakeholder my first question would be why there were so many bugs to fix in the first place. Because of this I'd suggest that in addition to reviewing the quality and value of the work you've done on the bug fixes (which others have provided useful guidance on) you also look at the root cause of the bugs.

You could take a random sample of your bugs and try to trace them back to their origin. Did the project have sufficient resource in place to deliver high quality code in the first place? Was enough time dedicated to quality assurance? Were the right development tools in place? Did the scope/requirements of the project change regularly or significantly? There's a lot of questions to ask!

Fishbone/Ishikawa diagrams or the '5 whys' are both useful techniques for getting to the bottom of problems like these.

Obviously you'll never eliminate all the bugs in a release but I think it's important to continually improve your processes to limit the amount of post-release bug fixing. Senior stakeholders will be happier that the initial releases work as expected/required and that your team can dedicate more of its time to further projects and releases.

Answer 3

I generally agree with the two comments posted so far. I can tell you two things that we have used these kinds of metrics for in the past.

1. Obviously, if you have one area that has a lot of bugs in it, you need to do something to improve development or QA procedures in that area.

2. You can use the bug closure rate during this phase of development to estimate future bug closure rates which will help you schedule future releases.

Otherwise, unless you have specific questions, it's tough for us to give you much more information.

Answer 4

I've been collecting bug related data for years now, and the number of fixed bugs haven't given me anything useful, because this data didn't tell me much about our projects of effectiveness. It simply says that how good we are at fixing bugs, and doing something we shouldn't have done in the first place.

When I talk about solving bugs I like to use an analogy to the police work: the purpose of policing should be preventing crime not chasing, which translates to analyse the circumstances of when certain bugs are appearing and make it sure that they less likely happen in the future.

So, my proposal is to watch the number of found bugs and where they have been found, by whom and when. Using this information you can put a distribution graph together and it will tell you which part of your application needs attention. The most problematic area is those bugs which were found by the customer (they are hard to reproduce, delivering the fixes are usually hard, and it is not good from the company's reputation point of view).

As a summary, I believe it doesn't matter how much bugs you fix, the number of found bugs matters and one should work on reducing this number instead of increasing the first.

Answer 5

Justifications are not Project Goals

Essentially, I want to be able to demonstrate the effectiveness of the recent time that we've spent dedicated to bug fixes.

This is a project smell that I usually associate with stakeholder justification for time spent on anything other than new features. However, it is important to note that justifying time spent on a task (regardless of what that task is) is a common organizational requirement, but is not directly related to any project goals.

While good project management is about managing time, scope, and budget, justifying the time spent on squashing bugs is only indirectly about those things. Instead, a project manager should be focused on communicating project status or identifying and controlling deviations from the project plan.

Addressing code quality is useful for addressing technical debt, and may be justifiable in terms of improved velocity, fewer customer-facing bugs, or producing something that can actually ship. However, if you haven't identified a problem that this bug-squashing task solves, it isn't really justifiable.

Consider this quote from Voltaire:

The best is the enemy of the good.

or the YAGNI principle to see why bug-fixing that's unrelated to project goals or requirements might be misguided, unless it's genuinely measurable technical debt.

Is Bug-Fixing on Your Critical Path?

All projects should have a goal. That goal may be "deliver a whatzit that eats irreplaceable data." If your whatzit data-eating feature is not working correctly, then of course you need to:

1. Fix the feature.
2. Identify the extra time (if any) as a deviation from your plan.
3. Refactor your plan to take the extra time into account.
4. Do some kind of post-mortem or retrospective to prevent such critical bugs in the future.

On the other hand, if your whatzit happens to electrocute users of the system, the risk has been identified to the stakeholders and accepted as a known risk by the organization---and if you don't have the moral gumption to quit over a customer-killing "feature---then your project road map should not include solving that issue.

Technical Debt is Bad

For me, it's not enough to just say that we were able to fix more bugs, I want to be able to show a dramatic overall improvement in the software.

This is clearly not a requirement from anyone outside your team, and thus is off the critical path. If you have a customer or stakeholder asking for specific improvements to some aspect of your software, then that's a legitimate (and likely measurable) requirement.

On the other hand, seeking "dramatic overall improvement" without an external requirement is called over-engineering. Most time spent on a project should be related to:

1. Meeting functional or non-functional requirements.
2. Reducing technical debt in order to make it easier to deliver on requirements.
3. Fixing actual defects that don't meet your requirements.
4. Preventing predictable future defects (see "reducing technical debt" above).
5. Following development practices that enable you to support your code in the future--see "technical debt" again, but don't over-engineer for potential problems that you can't measure or estimate right now.

Pithy Three-Sentence Summary

Fixing bugs is good. Preventing bugs is better. Over-engineering is best done as an intellectual exercise to impress your programming friends on a slow weekend.

Answer 6

Unfortunately, fixing bugs is like changing the oil in your car. It's an immediate expense that does not have an immediate return. Both activities result in cost avoidance, rather than a return.

Unlike an oil change, fixing bugs tends to begin reducing costs almost immediately. These returns are likely to include:

• Less effort to verify if a newly discovered bug is a repeat of an open bug.
• Lower cost to repair bugs as developers likely know where to fix them now. A couple of months from now they may have to spend much more time.
• Improved morale in the development team. The developers I know don't like leaving known bugs unfixed when repairing nearby code.
• Simpler code, as bugs don't need to be worked around. There may also to be a reduction in bug injection.
• Faster code production as developer can count on the existing code to work.
• Lower testing costs, as the testers don't have to avoid or work around the bugs.
• Improved product credibility. I quickly discount the quality of products when I find that known bugs were released. Typos also reduce the credibility of a product.
• For some bugs, there can be significant data cleanup costs to correct data that was corrupted by the buggy code.
• For bugs generating log data, lower cost of finding the cause of problems. This is also likely to include lower costs for storing and processing the logs.
• Lower costs for reviewing the open bugs list. These list should be reviewed frequently and the longer they are the more time it takes to review them.
• As other have noted, fixing bugs can be a learning activity. The sooner that practices leading to bugs can be corrected, the fewer bugs with their resulting costs.

Just managing the bug list can generate significant costs which may be the most measurable return. Many times it will merely delay some action by a minute or two. However, these minutes add up and most bugs will generate a lot of these delays. I've seen many bugs dragged along for months with the cost of keeping them for another month exceeding the cost of fixing them. The cost of verifying a long outstanding bug which got fixed by other changes can be quite significant.

As for a measurable metric, it is unlikely you will be able to accurately measure savings after the fact. If you had prior measures for some of the above avoided, you could look for attributable changes in your cost. You may see a decrease in the cost of ongoing development, but it may be with the normal variations. You may also get intangible improvements to team morale, team motivation, and the development process. Like the adds say, priceless.

Answer 7

Ultimately, the software exists to meet a customer need. That is the value of the software. The best way to make a case for bug fixes is to show how it increased the value of the software.

In the case of a software product, show how the bug fixes added to increased revenue.

In the case of internally deployed software show how it helped people do their jobs more efficiently, saving the company time/money and allowing the software users to focus on creating value for customers (instead of getting bogged down with buggy software).