Failed forecast: poor estimation or bad productivity?

Question

If the forecast of the team doesn't come true, how can I understand the reason for this?

It can be:

• Poor estimation during The Planning Meeting.
• Reduced productivity (for various reasons) during the entire Sprint.
• Unforeseen circumstances during the Sprint.

Only a subjective expert estimation can be applicable to this situation (in other words, talk with the developers, and ask why their forecast didn't come true)?

Or maybe some metrics, practice or tips exist for helping to detect the reason for this problem?

Answer 1

3rd reason why a forecast may not come true - the inherent uncertainty

You rightly used the word "forecast"! Prior to 2011 the Scrum Guide had the word "commitment". That caused two problems:

1. The development teams were under pressure to deliver on the "commitment" at all costs. Often this cost would be a drop in quality.

2. The Product Owner and the stake holders would make downstream planning, such as product launches and client commitments based on the hard "Commitment" by the development team.

In spite of the best efforts by the development team to estimate carefully and their best efforts to deliver on the commitment the work may not be completed.

In one of my previous projects we were building a secure application for online banking. We had selected an open source SSL toolkit for the encryption. We ran into a problem. Even though our lead developer was highy skilled and made every effort to make this functionality work or find a work around we could not do so. We identified that it was a bug in the open source software. Eventually our lead developer fixed the bug in the open source toolkit and contributed the bug fix back to that project. It was gratefully accepted. But, our project did take a hit on the time line.

You can read more about the background to the change in the Scrum Guide from "commitment" to "forecast" here:

Commitment vs Forecast: A subtle but important change to Scrum

However, it is possible in your team there is a pattern of poor estimation or reduced productivity, beyond this inherent uncertainty.

Only a subjective expert estimation can be applicable to this situation (in other words, talk with the developers, and ask why their forecast didn't come true)?

Your comment above seems to indicate a command-and control style of approach where someone from outside the team (expert) would monitor and control the team. The Scrum approach is for the team to identify the issue in the sprint retrospective and come up with ways to overcome it.

Answer 2

The OP does not really describe the context of what was wrong with the estimate or forecasts. By estimate, is the OP talking about a planning value or a probabilistic range. If the former, were the actuals within the probabilistic range and, if so, what was the degree of variance. If the latter, were the actuals way off the charts?

There are thousands of variables at play when actuals come in. Some are aleatory, random in nature and part of the process; while others are epistemic, specific drivers that cause actuals to come in high or low. Doing post mortems are excellent to try to discover what caused your variances; however, I would also argue that spending a ton of time trying to understand your variances when your actuals were well withing the normal range of probabilistic results is likely a huge waste of time. For example, if my estimate to complete this project was between eight months to a year, I targeted nine and half months but my actuals came in at a 11, who really cares what the drivers were that cause me to be late 1.5 months? What value would that provide me when estimate the next project? Eleven is comfortably in my range from eight to 12, my estimate. I do the project again, I could come in at 9.2; again I could come in at 10; again I could come in at 11.9. Understanding the drivers would be interesting but there are too many variables at play for that to make a difference.

It is important for the OP to put in context what he means by "wrong." Because simply missing your targets does not mean wrong. It just means you missed it this time.

Answer 3

First of all, reasons for the described mismatch are not limited to the two. I can imagine two additional and am pretty sure that a group brain storming will identify some more:

• bad estimation or if said more general: wrong estimation caused e.g. by uncertainty
• lazy team
• bad working conditions, e.g. caused by organisation (e.g. multiple interruptions) or technical (e.g. server down times)
• personal belongings, e.g. sickness or problems at home

But I agree with you regarding the search for the root cause: The simplest way to figure out what went wrong is to ask the team, e.g. during the retrospective.

If you don't want to do this or the team is uncertain, statistics might help you:

Assuming Theory Y, there must be circumstances hindering the team in delivering optimal work performance. Have a look on work performance data, e.g. time booked on the project. This will help to figure out if the real sprint length matched the expected length.

You might check the working conditions:

• Did a new construction site started in front of the window?
• Was it extraordinary hot?
• Was there a fire alarm / black out / black swan?

But finally, you should perform a root cause analysis:

Sit down with the team and ask what assumptions were made and which were wrong. You might use an Ishikawa diagram to figure out the root cause.

Answer 4

Scrum takes an empirical approach to estimation. It was realised that the process of asking teams to estimate and then relying on those estimates is fraught with problems. It is much more effective to allocate some work to a sprint, measure what gets done and then make predictions for future sprints based on those measurements.

The forecast of the team should be based on past performance. As long as they are consistently sizing stories then the past performance should help to predict what they are capable of in the future.

For example, say the team takes 30 story points in to a sprint, but only manages to complete 25. In the next sprint they may well reduce the number of story points they bring in to 25. The intention is to use a rolling average of the actual achieved results to determine future performance.

If your team is not getting all the work they committed to done in a sprint the answer is simple. Reduce the amount of work you commit to.

Even this approach does not guarantee successful comittment with every sprint. Development work has inherent unpredictability, particularly when a team is working closely with business users to ensure that the product meets their requirements. A Scrum team that allocates 20 points to a sprint and only achieves 16 would not regard this result as a disaster. They would discuss in the retrospective the reasons why not all the targetted work was completed and see if there are any ways to improve. And the velocity would be reduced to take in to account what has happened.