13

I've asked this question to few scrum masters: "How do you tell, that your team is improving? How do you tell that actions taken during retrospective result in real improvement in efficiency/productivity?"

Most answers I get are "when velocity grows". Or the opposite, when you see velocity fall, it is indication of worsening performance of the team.

Is there any basis behind this? Can velocity be used as a metric by which the team can tell if it is improving? Or is it simply case of using the most available metric instead of looking for metric that can be shown to be related to actual improvement?

One thing I'm worried about is that team can easily game this by simply estimating higher. So instead of task taking 2, they would give it 3. Creating an illusion of improving velocity.

  • Closely related: pm.stackexchange.com/a/17705/4271 – Todd A. Jacobs Oct 21 at 14:41
  • @ToddA.Jacobs Thanks. I did see that question. But I think the nuance is slightly different. The question could be reworded "If change is made to process or team, can velocity be used to determine if that change had positive, negative or no impact on efficiency or productivity?" – Euphoric Oct 21 at 14:48
  • 6
    I think the real issue you're dealing with is that velocity is a short-term detective control that has some utility as a planning value. It is not a productivity metric. Abusing it as such is a known anti-pattern. Search most of our velocity questions or the velocity tag for more on this topic. – Todd A. Jacobs Oct 21 at 15:10
  • No. Velocity doesn't tell you anything beyond "how many tasks in this sprint are done". – Davor Oct 22 at 23:03
23

Velocity is not a valid measure of team or process improvement.

Velocity, especially when it is based on Story Points, is extremely unstable. By itself, Velocity doesn't reflect the capacity of the team, but capacity will limit Velocity. It's also easily disrupted by changes to the team's way of working (which includes the definition of what a Story Point is), the team's composition, and an understanding of the problem at hand. As you point out, it's also easily gamed and its instability makes it even easier to game.

The only use for Velocity is in a very short window of time for a single team to figure out how much work can be brought into a single Sprint. Looking back too far can disrupt its usefulness. Using it as a predictive measure, especially too far into the future, is also error-prone. Trends are also error-prone.

  • 11
    What's not explicitly said in this answer is that if management starts using velocity as a metric for team or process improvement, then all the natural reactions to it will 'corrupt' the velocity metric, make it unusable for its primary purpose of sprint planning, and your sprint process will suffer because of this. – Peteris Oct 21 at 18:13
  • 6
    Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." – anaximander Oct 22 at 8:14
  • 1
    @Peteris: On top of that, it also implicitly incentivizes corner cutting. Half-assing a task takes less time and thus you get a much higher "points per time spent" output, whereas it's nigh impossible to really track which change caused which bug - or at least it takes more effort just to deduce it rather than just fix the bug regardless of who caused it. Either way, it costs more time and effort for the same end result. – Flater Oct 22 at 10:51
5

Your concern is absolutely correct. Velocity as a team improvement metric is problematic because an improving team will often see their velocity increase but just because the velocity increases doesn't mean the team is improving.

What you really want to look at is value delivered. There are a number of ways to assign value to work. If your team is already comfortable with relative estimation, an easy way to track value is to have stakeholders create a value score using relative estimation (most teams I know who do this use something like fibonacci * 100 - 100, 200, 300, 500 - just so the numbers are visibly different than story points). If you have good ways of quantifying actual revenue or cost savings, that's even better.

Of course, even with this, you need to be careful how much you stress metrics. If you focus on this metric (or any others) too much, the team can become myopically focused on that score and become short-sighted. This is just like companies that maximize this quarter's gains while sacrificing long-term health.

  • This method is just as ripe for abuse as measuring story points, only its the stakeholders who can pump the value metrics to make themselves look good. It also is more likely to capture changes in the value of stories than it is to capture changes in the team's capacity/performance/etc. – asgallant Oct 22 at 21:31
  • @asgallant: Any system is ripe for abuse from some angle (i.e. role). There is no perfect system or we'd all be using it. – Flater Oct 23 at 10:08
  • @Flater yes, true. That doesn't invalidate the point that this is a terrible metric for team performance, even ignoring the potential for abuse. – asgallant Oct 23 at 18:35
  • I think, perhaps, you and I are using the term stakeholders differently. I can't imagine what would possibly drive your users to inflate the team's numbers. – Daniel Oct 23 at 23:31
3

Hitting the nail on the head

One thing I'm worried about is that team can easily game this by simply estimating higher. So instead of task taking 2, they would give it 3. Creating an illusion of improving velocity.

This is a spot on recognition of the flaws of using velocity as a metric. Though also note that it's not always an intentional gaming of the system, it can also be a matter of genuinely bad estimates leading to a fake boost on the metric chart.

I do think this can be generalized, as any metric can be gamed. There isn't a single (ojectively measurable) metric that is impervious to being gamed. In writing this answer, I attempted giving you a better metric, and every single one I thought of had a way to game the system if the right people with the wrong intentions collaborate.


Without scrum

When you think about the pre-scrum period, the biggest problem with software development (from the point of view of a company and its management) were the missed deadlines due to unforeseen consequences of implementing changes, i.e. "we now realize we need to change B because we changed A. It'll take a few weeks extra".
Development ran past deadlines and non-developers had no way of gauging the completeness of a project as it was entirely dependent on the developers' understanding of the codebase and its pending tasks.

This led to developers having effectively free reign over the development process, and if you give someone sole reign over their own work assignments, of course they are going to err on the side of taking some extra time to complete a task, or to take on a nice-to-have under the guise of it being a necessity, or to focus on added value to the developers more than added value to the customer.
This doesn't mean it's always malevolent, it may simply be a matter of developers not thinking things through "because we can just say we need more time".


With scrum

To that end, the most important thing that scrum adds is not so much how quickly you can implement a change, but rather how much unforeseen consequences you've had to deal with.

If you challenge me to take only one metric of a team in order to approximate its quality, I'd mostly stick to comparing the absolute difference between estimated effort and real effort. The lower this difference is, the better your team is at estimating the incoming workload (If you notice a caveat here, I'll get that in a moment).

The amount of points a team handles in a sprint is less important than whether the team is able to stick to the points they expected to stick to. This goes either way: if the team took on a lot more than they expected to be able to, that's also indicative of bad judgment. People tend to think of this in positive terms ("they did even more than we asked them to!") but forget that this is indicative of the team being bad at estimating.

Developers don't magically work twice as fast on a good day. It's much more likely that some tasks were overestimated, which means your metrics that base themselves on these bad estimations are going to be way off as well.

Scrum by itself might not solve the productivity issues of your team when you are dealing with ill intent, but at the very least it documents the team's performance and gives more power to developer consensus (as opposed to rogue developers).


Caveat

There is one caveat here, which I can oversimplify/summarize as "insider trading". If developers have secretly agreed to pad their estimates and then ensure that they pad their work time to match this estimate, the above approach also wouldn't catch this.

However, this isn't really something that scrum is inherently built to handle, at least not when the estimaters and the implementers are the same people. It's an inherent flaw of giving someone the ability to (pre-emptively) define the standard by which they will be measured.
Scrum tries to minimize "rogue slackers" by having the estimates done through developer consensus, but this system doesn't work if all the developers are "collaborative slackers".

The opposite (letting an outsider set the standard) leads to different issues, e.g. management upholding a nonsensical metric or customers asking too much of your development team. When unrealistic standards are applied with enough pressure, this leads to developers cutting corners in order to avoid getting whacked with the hammer, which leads to the unmaintainable codebases that everyone hates (developers hate working with them, management hates that every change takes much longer than it needs to, customers hate the endless parade of bugs).

Having developers set the standards means that you give power to the (consensus of) developers to define the appropriate quality-vs-quantity balance.

Which brings me to my final point:


The best metric

The best metric for your team is whichever metric most accurately indicates something that the team is currently trying to improve.

If you grade your team on their velocity, you are implicitly incentivizing them to cut corners, as a half-assed task yields a lot of velocity for its relative time spent on it. If their half-assed task leads to a future bug, whose resolution they again half-ass (leading to another bug), then your metric will show that the team is working really fast while in reality they are spinning their wheels in the mud and not making any actual progress.

If, for example, your projects have bled their budget on bugfixing and maintenance issues, this means the code quality is lacking. Most commonly, code quality decreases by developers needing to cut corners, which tends to happen when their workload it too big.

However, if your team has had trouble with delivering features and instead have been getting caught in a process of no added value, then measuring velocity can be a valuable metric in order to get the team to focus on making sure they always add value for the customer.

But whenever you declare any metric as a target (i.e. a universal measure of team quality), you also make it a target for people who want to game the system to their benefit. There is no one metric that is both just and beyond reproach (in terms of being gamed) when it is publicly declared to be a target.

Do not use metrics as a way to spot issues in the team, use metrics to confirm the resolution of issues you have already observed in the team.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.