What are the right conclusions to draw from the success or failure of a Sprint?

Question

I recently encountered the opinion that, if a Sprint is successful due to statistical reasons (e.g. the team made big mistakes during planning but the team delivered its stories anyway), then that's as good as if success was achieved thanks to accurate estimation and informed commitment by the team. Hence, I'm wondering if this is actually in accordance with agile processes and the principles behind those processes.

The question may be broken down into two parts:

1. If the team finishes the sprint, it means that...
2. If the team fails a sprint, it means that...

When using an iterative, agile processes such as Scrum or Extreme Programming, what conclusions can be drawn from successful or unsuccessful Sprints?

Answer 1

TL; DR

If your Sprint was "accidentally successful," then the Sprint is still a success. It just means that future Sprints might not succeed without process change.

Why This is a Good Question

When using iterative, agile processes such as Scrum or Extreme Programming, what conclusions can be drawn from successful or unsuccessful Sprints?

This is a great question because it goes directly to the heart of what agile practices are about, and explores the nature of the explicit and implicit controls that underlie frameworks like Scrum and XP.

Defining a Successful Sprint

A successful Sprint generally meets all of the following criteria:

1. The Sprint Goal was met.
2. The Sprint Backlog was completed according to the "definition of done."
3. The Sprint Review demonstrated user-visible features that elicited constructive feedback from stakeholders.

However, the definition of success deliberately leaves out issues of velocity, story point estimates, internal processes, or scope changes approved by the Product Owner that don't compromise the Sprint Goal.

If your Sprint met the goals, you should probably declare success and address any continuous improvement issues within the Sprint Retrospective. For example, if your Sprint was successful but the team's process was not reliable or repeatable within acceptable variance limits, that doesn't mean the Sprint failed--it just means the next one might not succeed without process change.

Defining a Failed Sprint

This is actually harder to define, because a Sprint can actually succeed through failure. For example:

1. An abnormal termination by the Product Owner with a return to Sprint Planning can allow the organization to take advantage of an opportunity.
2. Failing early and performing a retrospective before returning to Sprint Planning can represent a cost savings over failing to deliver, or over delivering the wrong thing.
3. The Sprint Goal was not met, but the organization extracted value from the process-improvement opportunity.

Again, note what is deliberately left out of the definition. A Sprint Goal can be met even if the Sprint Backlog is not completed; by itself, an incomplete Sprint Backlog does not mean the Sprint failed. Likewise, a Sprint Review that doesn't delight stakeholders just means that the iterative feedback loop of the Sprint is operating as designed.

Of course, if the Sprint Goal was not met, and there's no silver lining for the team or the organization, then it's fair to say that the Sprint has failed. However, even a failed Sprint is rarely a total loss; there is usually at least some residual value from the iteration, allowing even "failed" Sprints to provide some marginal benefit.

Drawing Conclusions From a Single Failure

Analyzing failures is one of the purposes of a Sprint Retrospective. It offers an opportunity to conduct a post-mortem and identify potential improvements to the team's process.

From an overall project point of view, though, an isolated failure is just a single data point. Until you have a pattern of failures, the only conclusions you can reasonably draw are:

1. You have a potential process problem that should be carefully reviewed.
2. You have missed an opportunity to extract value from the process by failing early.
3. You may need to re-estimate your Product Backlog and release plan if you have insufficient slack in your project to allow for variance between iterations.

Of course, if you really do have a pattern of failures, then it's up to the team and the stakeholders to analyze why the process broke down repeatedly, and whether the process or the project can be salvaged.

Answer 2

I cannot speak to the principles behind agile processes but from a general perspective I opine that the results we achieve from work--or any action really--are mostly influenced by stochastic variables. In other words, we have less control over our results than we like to believe.

When we do post mortems of a recently finished effort, when things go well we like to attribute those results to our great decisions and levers we chose to pull when we chose to pull them. Similarly, when things go poorly, we do the opposite and identify all those things over which we had no control. Misattribution and Illusion of Control govern these thought processes and behaviors.

I think it is valuable to identify those things that influenced a good result; it would be far too fatalistic not to do so. However, a deep analysis of those variables needs to include some balance with what happens in a random way. Otherwise, we end up with a ton of false beliefs of cause and effect where there really is none. And there are a ton of examples of this in business.

Answer 3

In my opinion, "success due to statistical reasons" as described in your question is not as good as success that is achieved because of accurate estimation and informed commitment because the former cannot be reliably reproduced. In fact, "success due to statistical reasons" is likely more dangerous in the long run because it will mask foundational problems in your planning practices.

By way of analogy, it is possible to pick a winning football team even though your analysis is deeply flawed. This will be a "success" because you win some money this week. But in the long run you will lose money compared to the person who picks their team using a robust analysis.