1

Intro

I like to setup KPIs for my engineering/QA teams. I use that to evaluate performance and help determine bonus/raise each quarter.

This is one KPI I set for my engineers:

  • % of tasks that passed QA with no bugs

    • ie depending on the QA flow (per issue or per sprint), we see how many bugs were reported against all the issues that were completed by 1. the engineer that fixed the issue 2. the other engineer that peer reviewed it.
      • ie in a given sprint engineer X resolved 20 issues, which got reviewed by engineer Y. Assuming the QA does their work on a sprint basis, and that the QA found 5 bugs in those 20 issues, the tasks that passed QA with no bugs percentage is 75%
  • % of tasks that functioned properly post deployment Same idea as above, but this one will be treated for code that actually gets released live and is used by users

This is one KPI I have for my QA staff:

  • number of bugs found per issue
    • ie the more bugs a QA person finds in an issue the better.

Problem

The KPIs for the engineers and QA personnel seem contradictory, or said another way: they seem like a zero sum game. Which obviously isn't good for employee morale and would cause rivalry/resentment.

How can I change this so that both engineers and QA staff have KPIs that make them work together as opposed to against each other?


update: a concrete case

this is in response to novigt's answer below as well as in reference to this comment on a related post:

Are the testers involved before a build? Meaning are they involved in developing the requirements or use cases or user stories, reviewing design documentation, or participating in code reviews?..

I figured that proper and sound documentation of scenarios is a huge part of evaluating QA personnel (keep in mind here I'm talking about non-technical QA who simply do black box testing and cannot do automated tests etc).

I agree that a metric that calculates total number of bugs is bogus, and i agree with the incidents in production metric.

Let me explain my new metric with an example:

enter image description here

Suppose I have a single login screen (forgot password/remember me etc not included for brevity) that I would like to build and deploy. Using Gherkin Syntax the QA engineer writes the following scenarios:

Scenario 1: Happy path login
given user enters correct username/pwd
when user clicks on login
then user should be directed to dashboard

Scenario 2: login failure
given user enters false username/pwd
when user clicks on login
then error message shows: incorrect username/pwd

this documentation is given to the engineers (along with the designs) on sprint one , who develop it, peer review it, send it back to QA for testing and then is deployed in version 0.0.01 of the app.

Once on prod, the following bug happens

given user enters correct username/pwd
and the server is down when user clicks on login
then
expected
user should get an error message: service is currently unavailable
actual
a spinner spins for ever


Here are the KPIs/metrics we capture for this scenario:

  • number of scenarios: 2
  • number of incidents in production: 1

metrics as a snapshot don't give the whole picture, so let's go on:

so on sprint two, the QA staff updates the documentation with scenario 3:

scenario 3: handle login when server is down
given user enters correct username/pwd
and the server is down
when user clicks on login
then user should get an error message: service is currently unavailable

the engineers fix that bug, and deploy it on release 0.0.02, on the release this bug happens:

given user enters correct username/pwd
and the user is not connected to the internet when user clicks on login
then
expected
user should get an error message: user is offline
actual
a spinner spins for ever


Here are the KPIs/metrics we capture for this scenario:

  • number of scenarios: 3
  • number of incidents in production: 1

  • rate of increase of scenarios: 33%

  • rate of decrease of incidents in production: 0%

so we use the last two KPIs (increase in scenarios and decrease of incidents in production) together to evaluate the performance of a QA person. The final evaluation should use both KPIs (in addition to other metrics that can be discussed elsewhere) to evaluate an engineer. This way increasing useless case scenarios alone won't be positive alone, and also an increase in incidents in production without a corresponding increase in scenarios should be a red flag for the engineer. I know one can also achieve this by having comprehensive unit tests that have >95% code coverage etc.. but let's assume we're not there yet.

Is there a problem with this?

  • 3
    "Once on prod, the following bug happens" This is not a bug. A bug is a deviation from the spec. This was never in the spec to start with. Labelling this as a bug will make your Engineers very unhappy. A bug is a mistake I made as an Engineer, something not working to spec. Something not in the spec is a feature request and should never be held against anybody but the one writing specs. – nvoigt Jan 3 '17 at 11:53
  • @nvoigt fair enough i won't call it a bug, i'll call it spec miss.. and so a KPI for QA staff (again, not engineers) is the amount of spec misses per sprint.. works? – abbood Jan 3 '17 at 15:02
  • Who makes the judgement that the QA should have specced that? And if that person knows what should have been specced, why didn't s/he do it in the first place instead of letting the poor QA guys guess? – nvoigt Jan 3 '17 at 15:13
  • It's the QA's job to ensure that documentation covers all possible user cases. This doesn't mean that all those use cases will be implemented in release 0.0.01, but we at least need to know they exist (they can be done in a later release as per the lean model). Further this isn't an all or nothing. I'm not saying that if a spec miss happens the QA will get beat up in the parking lot. I'm saying that these are metrics we can use as a supplement in their evaluation... – abbood Jan 3 '17 at 15:21
  • .. Ie when I evaluate an employee I look at several things, including their KPI's. You are making an absolute statement that all metrics/KPI's are bad? Let me try to see things more from your perspective. Assuming that you have knowledge workers reporting to you, how do you evaluate their performance (with the absence of any metric)? And how can you back up that evaluation? How can you scale that process? – abbood Jan 3 '17 at 15:21
4

So right now, as an Engineer, I take a single task, mull over it for 4 weeks, make sure it contains no bugs at all and I am the best worker you ever had. Despite the fact that it took me 4 weeks to deliver such a simple task.

On the other hand, you have QA, who is wildly dependent on getting a crappy Engineer assigned. The crappier the engineer, the more rewards the QA will get. What happens if I have to test a product from a good Engineer? Will I be put on a performance improvement plan for not finding enough bugs?

Your metrics are arbitrary and less than useful. Your people are not worker drones, they use their intelligence to get their job done. So don't insult them with metrics that can easily be gamed.

At least start with what really matters: incidents in production. Bugs that escaped both Engineers and QA. Try to minimize those and you will see that Engineers and QA are actually a team working towards the same end goal.

However, it's debatable whether having metrics is worth it for intelligence workers. Nobody has found the right ones yet and the wrong ones can backfire easily. If you base anything on your KPIs (like praise or salary) be prepared that people will be clever enough to outsmart you on any of them. Then you are stuck with people that did their jobs, but look average on the metrics and people that gamed the system instead of doing a good job, but look great on your system.

  • 1
    Your point is well taken about the initial kpi being not useful, I updated my question with a different metric that hopefully addresses your points. Please take a look. Also: it's debatable whether having metrics is worth it for intelligence workers. Nobody has found the right ones yet can you please mention a research or statistic (or even some opinion piece) that supports that point? – abbood Jan 3 '17 at 11:46
  • @abbood No I cannot back something up that does not exist. I have never seen KPIs that worked well for knowledge workers, nor heard from them. If you find some, let me know, I'd be happy to have them. – nvoigt Jan 3 '17 at 11:55
  • @abbood Joel has (had, but I'm not aware of changes) a similar outlook on KPIs for knowledge workers. – nvoigt Jan 3 '17 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.