2

I am trying to overhaul the bug-ticket management process for the company I'm working at. One of the requirements is that top-level management knows how productive the developers are when fixing bugs. We are in a critical R&D restructuring phase and need to protect the hard workers from work overload caused by demotivated developers. To understand these individual cases we need numbers.

Now my problem is: I am planning to consider metrics like number of tickets solved, median duration, difficulty of bugs, bug lifecycle time, time until first reaction, number of open requests, etc. Maybe a point system. The difficulty and severity is rated by the developers themselves as they are the closest to the code. But of course the management is rightfully skeptical about these metrics because it is not hard to game them.

Are there better ways to get more objective results? Of course I'm going to include the developers, and managers in how to manage this issue, but first I need to know if there are better ways that I don't see yet. Metrics that are harder to game because they more directly connect to the skill and motivation level of the developer, metrics that would better show the workload they should be able to handle and the actual workload handled...

13
  • 5
    This looks like an XY Problem. You are asking about a solution, but what is the problem you are trying to solve?
    – Bogdan
    Oct 18, 2020 at 11:37
  • 2
    Welcome, Jasper! In addition to any answers you might get here, you could also have a look around some of the questions asked at Software Quality Assurance & Testing Bugs and metrics are a common topic over there.
    – Llewellyn
    Oct 18, 2020 at 12:46
  • 2
    If your question is coming up with kpis that cannot be gamed, I have yet to see anyone come up with any that are at the same time meaningful and gaming resistant. If you need a development team big enough that you cannot eyeball the relative performance, your main problem is probably that your development team is too big. Oct 18, 2020 at 12:46
  • 3
    Recommended reading: martinfowler.com/bliki/CannotMeasureProductivity.html
    – Bogdan
    Oct 18, 2020 at 13:36
  • 2
    There is absolutely no need to have quantitative measures of productivity and performance. Why do you think that you need this?
    – Thomas Owens
    Oct 18, 2020 at 14:03

2 Answers 2

3

TL;DR

Successful leaders measure project outcomes rather than individual productivity. Measuring individual productivity is generally an anti-pattern that obfuscates deeper structural problems.

Do you have too many bugs? Turn your teams loose on reducing the amount of bugs released into production. Are bugs taking too long to fix? Get your developers and testers involved in improving code coverage and diagnostic value of your test suites. Think your developers or testers are "lazy?" Make sure it's not the process that's broken, or unrealistic expectations from outside the team at fault; then hold management accountable for hiring inexperienced or ineffective people, or lacking the leadership to redirect or kill a failing project.

Metrics are useful for process improvement. They are rarely accurate measures of individual productivity, and are often poor proxies for determining accountability. In that regard, your mileage will not vary.

Analysis & Advice

Metrics can be helpful, but in knowledge work (and especially in software development) measuring the right things is NP-hard. It often grows from a desire to measure by proxy, and is therefore always a leaky abstraction that can inherently be gamed.

Especially when evaluating "bugs," you can't accurately measure in a non-complex way. You can't simply measure number of tickets closed or lines of code touched for a patch. For example, measuring the complexity of a reported bug, time needed to isolate or replicate the bug, and determining the cyclomatic impact of the bug and/or patch on the rest of the code base are a priori data points needed to perform any sort of apples-to-apples comparison. While there are people who study this sort of problem, the pragmatic view of those in the industry is that the juice is almost never worth the squeeze.

Imagine a bug that takes two weeks to track down, but only one character of code to fix. Is that developer more or less "productive" than one who fixes a bug that takes only two hours to fix by removing a dozen custom classes and replacing it with an off-the-shelf component? If you can't answer that question is terms of anything other than time, then you've failed to fully capture the complexity of the abstractions here.

The only pragmatic approach to determining individual developer productivity is to ask the other developers on the team to evaluate one another. Experienced, self-organizing teams will generally know how hard the bugs are, why certain classes of bugs crop up routinely, and whether each team member is contributing as effectively as possible within the limitations of the current process.

Be aware that asking teams to measure individual performance, rather than simply measuring team output, invites process and structural problems that can be very hard to fix. That isn't to say that some people aren't more efficient or effective than others, but unless a person's performance is disrupting the team or process, then looking at individual performance is usually a sign of Theory X management. Measuring individual rather than team productivity will generally encourage CYA behaviors rather than teaming or continuous process improvement.

2
  • Thanks. This was really helpful.
    – Jasper
    Oct 19, 2020 at 22:59
  • Could you recommend literature or any ressources for further reading?
    – Jasper
    Oct 19, 2020 at 23:07
0

First, you have to start by categorizing the bugs by what caused them. There are different kinds of bugs, including, but not limited to:

  • Logic error
  • Out-of-bounds condition
  • "Works by me", the developer, but not in production
  • Typos and grammatical or translation errors
  • Works on certain platforms, but not on others
  • No longer works on older/newer versions of the platform
  • "QA bugs" - if you do a, b, & c (usually very quickly) then weird things happen.
  • GUI bugs: Something doesn't look aesthetic
  • "I don't like the way this looks"

Note that as you go down the list, the blame is less & less on the original developer.

Point being, that you can't treat these bugs the same way. At some level the first few bugs are the fault of some programmer, the last few have PM (or even nobody) to blame.

You even have bugs (QA bugs) that may be pointless - albeit fun/challenging - to fix.

Then you can categorize the bugs (again) by difficulty in solving them. Clearly fixing a typo is close to trivial, while upgrading the code to a new platform and/or making it backward compatible is extremely challenging.

You can then start figuring out:

  • Who is causing the bugs. These programmers may need training of some sort.
  • Who is fixing what type of bugs. Who always tackles the difficult ones and who grabs the easy ones.
  • Are the easy bugs being fixed at a (much) higher rate than the hard ones.
  • Which bugs go back & forth to QA; who doesn't know how to fix bugs properly - or breaks something while fixing something else.
  • Etc. Stare at the raw data and find patterns.

By slicing and dicing the bugs base based on (pseudo-)facts it's harder to game the metrics and you get a better picture of who is working hard and who is pretending to, as well as who is causing the bugs in the first place.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.