1

The product manager of my development team wants to use this ratio as a KPI, which allegedly will indicate things like:

  • the quality of our code
  • the amount of focus we put on development before considering a task done
  • how much room for efficiency-improvement we have

He expects a ratio no bigger than 0.15 (15% max of bug cards in an Epic)

Last bit of information: We Make Games!

Is this a good metric to watch for? I've heard 15%-20% is a healthy ratio to expect in "common" software development but does this undoubtly applies to Game Development aswell?

1
  • That's a fairly arbitrary metric, and while I think there's a good question in here I think it's an X/Y problem that's leading you to create an "opinion poll" question. Those are generally off-topic here, but I think the underlying topic is fine; the question just needs some TLC so that it isn't inviting a discussion or answers that are subjective or all equally valid.
    – Todd A. Jacobs
    Commented Nov 11, 2022 at 21:36

5 Answers 5

3

TL;DR

You have one or more process problems that have been abstracted away into a proxy metric that is being used to "hold people accountable." This is almost always counterproductive, but is common in many company cultures. You will need to work as a team (including the product manager) to apply systems thinking to find the real problem, the right solution, and non-proxy metrics to measure your progress along the way.

Fiat Metrics Often Measure the Wrong Things

He expects a ratio no bigger than 0.15 (15% max of bug cards in an Epic)

Great! When he can run the process, write the software, perform the testing, and all the other stuff that goes along with that stuff himself, then he can set any fiat targets he likes. Until that happens, though, he needs to collaborate with the people performing the work to identify an acceptable level of quality and a measurable (and ideally executable) Definition of Done.

Besides being an arbitrary management target, it's not tied to anything actionable. You can measure it, sure. However, such metrics don't lead to actionable insights into the process or the underlying issues in what is essentially an X/Y problem.

Measure Outcomes Instead of Proxy Metrics

Aside from being more collaborative, this person needs to stop using proxy metrics and identify testable key results for the product and the process. Measuring "bugs" (which is sort of a hand-wavy term to begin with) is generally a proxy for "waste" in the Lean sense.

Essentially, this metric is saying:

Let's aim to waste 15% or less of our project capacity!

That's rather arbitrary, especially if the goal is lower than the industry standard. I won't argue whether or not the numbers are even valid since the metric itself is flawed.

Instead, this person should be measuring the outcome or key result that they're chasing. For example, if the project is over-budget because more time is spent fixing bugs than developing new features, you need a budget or labor metric rather than a proxy metric that counts bugs.

Likewise, if the company is losing money because of excess returns due to quality control issues or an inability to meet customer demand, then the metrics should be measuring returns or fulfillment issues directly rather than through a proxy.

Use Systems Thinking and Fix the Process

Bugs don't just happen. Even if you accept the premise that the percentage of bugs is inherently a problem, you can't measure your way out of a problem you don't have a root cause analysis for. In other words, demanding fewer bugs won't solve the problem; the way to fix the number of bugs is to fix the process that's allowing the bugs to happen or escape into production.

As a practical example, if the product manager and the project sponsor won't spend money on CI/CD tools or give the team sufficient slack in the process to perform test-first development, then bugs are inevitable. Bugs are prevented, caught, triaged, and more easily fixed if:

  1. Testing is baked into the design.
  2. Testing is automated.
  3. Every bug results in a new regression test to make sure it doesn't recur.
  4. Tests provide "living documentation" for the product's features and the Definition of Done.
  5. The product's architecture and development processes treats testing as a first-class objective.
  6. Budget, time, equipment, and labor are all made available to make quality control a first-class objective.

The majority of the time, this topic comes up because the budgetary authority or the person prioritizing the work has explicitly de-prioritized planning, testing, or tooling. These essentials are then replaced with human labor rather than automation or effective quality controls, and exhortations to "work smarter, not harder" or "hold people accountable." The latter almost always gets applied to the project team rather than the person actually accountable from a RACI model perspective for the budget, scheduling, prioritization, and other business-focused goals of the project.

I'd bet money that's the case here. Whether or not that's true, the whole team needs to carefully re-evaluate the process to find out why expectations aren't being met or excess labor is spent on fixing things post facto.

Dig for a root cause, which is almost always a process problem. Techniques like "The Five Why's" can often be helpful, but the point is to look at the whole process and drill down as far as you can, rather than just taking the issue at face value. Then the product manager and the development team need to collaboratively inspect-and-adapt until the root cause is resolved in a satisfactory and sustainable way.

6

That seems a very formal and ultimately futile way of viewing things.

Right now, your rate is X. Whatever ratio others tell you is good, that will not change your current reality.

And if you say Y is a good rate, then people will stop at Y. Why stop there? Why stop improving after an arbitrary number?

Instead, why don't you measure the current rate and then go into an open discussion with all participants, how to improve your whole process so that this rate goes down.

"Bug rate" is very rarely just a result of developers making mistakes. In my last 20 years, I would say about 10% of what I was presented with as bugs were actually my programming mistakes. The rest were insufficient requirements, changing requirements camouflaged as "bugs" to shift blame and responsibility, insufficient testing despite me asking for more (either more time or more automation) and arbitrary deadlines pushed through, where "done with bugs" is better than "not done yet" for some upper management guy on paper - regardless of the fact that the "bugs" are actually missing functionality, rather than mistakes a programmer made on existing functionality.

So if you want to improve that, you need to improve the process. All of it. As a KPI of software development it's great. As a KPI for software developers is completely pointless, because it's not in their sole domain to change this.

Measure it. Then improve it. It does not really matter what others achieve there, because what you consider a "bug" is not a globally valid constant. It is important, that your team improves. Whether it improves from 20% to 10% or from 40% to 20% is just meaningless numbers. It measurably improves. That is the important part.

1
  • 1
    I fully agree. The ratio itself is not an indicator of anything. It does not clarify the source of the bugs, and it does not clarify the magnitude of the bug. And then: how do you measure the magnitude of a bug?
    – virolino
    Commented Nov 10, 2022 at 10:14
2

Additionally to @nvoight's answer, I want to say that there are other "KPI's" which would be a lot more helpful:

  • the rate of the appearance of bugs (per day, per release...)
  • the rate of closing the bugs (per day, per release...)
  • the cost of solving the bugs
  • the histogram (on a time + release axis) of "implementing" the bugs
  • the root cause of the bugs (pie chart): requirements missing or wrong / incomplete, programming, bad testing, feature not implemented but tested etc.

Once all these kinds of "KPI's" are in place, some analysis can be done in order to improve the entire work-chain.


Why [bug cards]/[total cards] ratio is useless

As I mentioned before, the following are impossible to define and measure:

  • the magnitude / complexity of a bug
  • the magnitude / complexity of other tickets

Example: Both actions below have the "magnitude" of one card:

  • change the background color from X to Y
  • implement an entirely new communication stack

Now consider that you have one "bug card" for each actions. Does it mean that the two actions were done at the same level of quality (ratio 1/1)?

Actually, no. The background change was a fully failed task, while the comm stack task has "just" a minor "inconvenience". Unless the bug is "comm stack actually missing" :)

1

Instead of looking for a custom ratio of a dubious KPI, I'd bring to the table the observation of worldwide known metrics. In this case, DORA (https://www.devops-research.com/research.html) is a case in point.

One of the 4 key metrics it tracks is Change Failure Rate. Worth look at it.

1

The issue I have with this metric is that it treats all bugs as the same. A bug might take an hour to fix or it might require a rearchitecting of your entire system.

If you are going to measure the ratio it would be worth also having a metric that measured bug fix cycle time.

The conversation would then be about a combination of the two metrics:

This month the bug ratio was a little higher than before, but we also reduced the bug fix cycle time, so that doesn't seem to be a problem

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.