Is velocity a valid measure of team and process improvement?

Question

I've asked this question to few scrum masters: "How do you tell, that your team is improving? How do you tell that actions taken during retrospective result in real improvement in efficiency/productivity?"

Most answers I get are "when velocity grows". Or the opposite, when you see velocity fall, it is indication of worsening performance of the team.

Is there any basis behind this? Can velocity be used as a metric by which the team can tell if it is improving? Or is it simply case of using the most available metric instead of looking for metric that can be shown to be related to actual improvement?

One thing I'm worried about is that team can easily game this by simply estimating higher. So instead of task taking 2, they would give it 3. Creating an illusion of improving velocity.

Answer 1

Velocity is not a valid measure of team or process improvement.

Velocity, especially when it is based on Story Points, is extremely unstable. By itself, Velocity doesn't reflect the capacity of the team, but capacity will limit Velocity. It's also easily disrupted by changes to the team's way of working (which includes the definition of what a Story Point is), the team's composition, and an understanding of the problem at hand. As you point out, it's also easily gamed and its instability makes it even easier to game.

The only use for Velocity is in a very short window of time for a single team to figure out how much work can be brought into a single Sprint. Looking back too far can disrupt its usefulness. Using it as a predictive measure, especially too far into the future, is also error-prone. Trends are also error-prone.

Answer 2

Your concern is absolutely correct. Velocity as a team improvement metric is problematic because an improving team will often see their velocity increase but just because the velocity increases doesn't mean the team is improving.

What you really want to look at is value delivered. There are a number of ways to assign value to work. If your team is already comfortable with relative estimation, an easy way to track value is to have stakeholders create a value score using relative estimation (most teams I know who do this use something like fibonacci * 100 - 100, 200, 300, 500 - just so the numbers are visibly different than story points). If you have good ways of quantifying actual revenue or cost savings, that's even better.

Of course, even with this, you need to be careful how much you stress metrics. If you focus on this metric (or any others) too much, the team can become myopically focused on that score and become short-sighted. This is just like companies that maximize this quarter's gains while sacrificing long-term health.

Answer 3

Hitting the nail on the head

One thing I'm worried about is that team can easily game this by simply estimating higher. So instead of task taking 2, they would give it 3. Creating an illusion of improving velocity.

This is a spot on recognition of the flaws of using velocity as a metric. Though also note that it's not always an intentional gaming of the system, it can also be a matter of genuinely bad estimates leading to a fake boost on the metric chart.

I do think this can be generalized, as any metric can be gamed. There isn't a single (ojectively measurable) metric that is impervious to being gamed. In writing this answer, I attempted giving you a better metric, and every single one I thought of had a way to game the system if the right people with the wrong intentions collaborate.

---

Without scrum

When you think about the pre-scrum period, the biggest problem with software development (from the point of view of a company and its management) were the missed deadlines due to unforeseen consequences of implementing changes, i.e. "we now realize we need to change B because we changed A. It'll take a few weeks extra".
Development ran past deadlines and non-developers had no way of gauging the completeness of a project as it was entirely dependent on the developers' understanding of the codebase and its pending tasks.

This led to developers having effectively free reign over the development process, and if you give someone sole reign over their own work assignments, of course they are going to err on the side of taking some extra time to complete a task, or to take on a nice-to-have under the guise of it being a necessity, or to focus on added value to the developers more than added value to the customer.
This doesn't mean it's always malevolent, it may simply be a matter of developers not thinking things through "because we can just say we need more time".

---

With scrum

To that end, the most important thing that scrum adds is not so much how quickly you can implement a change, but rather how much unforeseen consequences you've had to deal with.

If you challenge me to take only one metric of a team in order to approximate its quality, I'd mostly stick to comparing the absolute difference between estimated effort and real effort. The lower this difference is, the better your team is at estimating the incoming workload (If you notice a caveat here, I'll get that in a moment).

The amount of points a team handles in a sprint is less important than whether the team is able to stick to the points they expected to stick to. This goes either way: if the team took on a lot more than they expected to be able to, that's also indicative of bad judgment. People tend to think of this in positive terms ("they did even more than we asked them to!") but forget that this is indicative of the team being bad at estimating.

Developers don't magically work twice as fast on a good day. It's much more likely that some tasks were overestimated, which means your metrics that base themselves on these bad estimations are going to be way off as well.

Scrum by itself might not solve the productivity issues of your team when you are dealing with ill intent, but at the very least it documents the team's performance and gives more power to developer consensus (as opposed to rogue developers).

---

Caveat

There is one caveat here, which I can oversimplify/summarize as "insider trading". If developers have secretly agreed to pad their estimates and then ensure that they pad their work time to match this estimate, the above approach also wouldn't catch this.

However, this isn't really something that scrum is inherently built to handle, at least not when the estimaters and the implementers are the same people. It's an inherent flaw of giving someone the ability to (pre-emptively) define the standard by which they will be measured.
Scrum tries to minimize "rogue slackers" by having the estimates done through developer consensus, but this system doesn't work if all the developers are "collaborative slackers".

The opposite (letting an outsider set the standard) leads to different issues, e.g. management upholding a nonsensical metric or customers asking too much of your development team. When unrealistic standards are applied with enough pressure, this leads to developers cutting corners in order to avoid getting whacked with the hammer, which leads to the unmaintainable codebases that everyone hates (developers hate working with them, management hates that every change takes much longer than it needs to, customers hate the endless parade of bugs).

Having developers set the standards means that you give power to the (consensus of) developers to define the appropriate quality-vs-quantity balance.

Which brings me to my final point:

---

The best metric

The best metric for your team is whichever metric most accurately indicates something that the team is currently trying to improve.

If you grade your team on their velocity, you are implicitly incentivizing them to cut corners, as a half-assed task yields a lot of velocity for its relative time spent on it. If their half-assed task leads to a future bug, whose resolution they again half-ass (leading to another bug), then your metric will show that the team is working really fast while in reality they are spinning their wheels in the mud and not making any actual progress.

If, for example, your projects have bled their budget on bugfixing and maintenance issues, this means the code quality is lacking. Most commonly, code quality decreases by developers needing to cut corners, which tends to happen when their workload it too big.

However, if your team has had trouble with delivering features and instead have been getting caught in a process of no added value, then measuring velocity can be a valuable metric in order to get the team to focus on making sure they always add value for the customer.

But whenever you declare any metric as a target (i.e. a universal measure of team quality), you also make it a target for people who want to game the system to their benefit. There is no one metric that is both just and beyond reproach (in terms of being gamed) when it is publicly declared to be a target.

Do not use metrics as a way to spot issues in the team, use metrics to confirm the resolution of issues you have already observed in the team.