2

The planning fallacy appears especially frequently when estimating tasks in software development. However agile estimation methods like planning poker etc. don't seem to be designed to avoid this problem.

So what do you do to avoid the planning fallacy when estimating tasks or tickets in IT projects? Are there any product management methodologies or best practices which are designed to avoid this common fallacy?

  • Can you explain in which way planning poker doesn't help with this bias? Or how you've used it? For me, it does deal with it fairly well, but that probably means we play it differently from you. – Erik Oct 15 '18 at 15:27
  • As far as I know planning poker does not make use of any of the suggested countermeasures mentioned in the article: en.wikipedia.org/wiki/… . I think planning poker is more designed to foster communication, but not for avoiding the planning fallacy. – asmaier Oct 15 '18 at 15:44
2

Planning Poker is designed to use (or at least can be used so that it involves) a form of Reference Class Forecasting.

When estimating with planning poker, you can pick a sort of 'baseline' ticket that represents about 1 point of work, and then estimate all other tickets as being "about twice as hard" for 2 story points, or "probably 10 times as hard" for 13 story points, or "a lot easier" for 1/2 story point.

By estimating from a baseline ticket and only ever planning based on comparison to that one, all estimates become relative. This should eliminate a fairly large amount of bias; you aren't thinking in time but comparative size, which is something humans are much better at.

Then, to decide how far you can get, you look at the team's track record for "story points per sprint" and fill up to something comparable. If the team gets better, they keep estimating the same way, but the "story points per sprint" goes up, so the amount of work they take on grows as well, without ever having to re-calibrate.

This method works as long as you keep reminding people that story points are not measures of time, but of complexity to a reference point, and that they're not allowed to make the conversion between the two at any time. It stops working the moment you decide that a story point is an hour, or a day, or any other measure of time, as you'll immediately fall back into all the estimation traps that you (and the article) mentioned.

  • +1/-1 (Net: 0). Planning poker is certainly a reference class system, but it primarily solves for the anchoring problem. It only controls for optimistic or pessimistic planning indirectly, and requires that a full panoply of cross-functional viewpoints be available and successfully integrated into the estimate. It's actually velocity (technically, the "forecast" in Scrum) that compensates for capacity or time-based bias, and then only to the extent that the process is applied consistently across Sprints. Planning poker and velocity work best together IMHO, but your mileage may vary. – Todd A. Jacobs Oct 17 '18 at 14:01
  • FWIW: An answer the stakes out the argument that the conversations around planning poker, uncovering the assumptions and perspectives of the Development Team, would certainly earn an upvote from me as mitigating planning bias more directly. But without a post-hoc metric like velocity, I'm not sure how you'd ever determine how effectively the process was at minimizing bias. Whether or not that's important in practice, it's the core of the question asked. – Todd A. Jacobs Oct 17 '18 at 14:05
  • I think to make Planning Poker into a form of Reference Class Forecasting the baseline ticket must be a task already finished, right? Because only in that case complexity and time needed are really known for sure and can be used as reliable reference. – asmaier Oct 17 '18 at 16:54
2

You cannot "avoid" bias. It's always there. You can only minimize it by estimating a task using multiple different methods and focusing on probabilistic estimates versus deterministic estimates. And, even then, you will only know how well you worked through the biases when you are able to compare your actuals with your planned values and you have a reasonably large number of comparison observations. This assumes you have a highly reliable method in tracking and controlling your actual values, and that is a huge assumption that is likely not very true.

As is true with all human biases, the more you think you have it under control, the more biased you are.

  • "As is true with all human biases, the more you think you have it under control, the more biased you are." I disagree strongly with this and how absolute you make it sound. – Venture2099 Oct 16 '18 at 9:38
  • Have you put in any time studying it? If we were able to control our biases, why would the scientific method be necessary to study something? And even with that we risk experimenters bias in the results. – David Espina Oct 16 '18 at 10:31
  • As a former Intelligence Analyst, we did a little bit of bias training..just a tad. – Venture2099 Oct 16 '18 at 10:40
  • I suspect more than a tad. Then I'm baffled how you would disagree with my statement. I haven't come across any study that shows our ability to control our biases with any degree of success or longevity. If you have something, please share. Happy to learn something new. – David Espina Oct 16 '18 at 10:56
  • 1
    There’s a sort of Dunning-Kruger effect in bias. I think bias is often implicit, and whether or not you can control the bias itself, you can certainly control for it. +1 for addressing bias mitigation as a more likely alternative than eliminating it. – Todd A. Jacobs Oct 16 '18 at 23:45
1

TL;DR

Per the “planning fallacy” Wikipedia entry:

The planning fallacy...is a phenomenon in which predictions about how much time will be needed to complete a future task display an optimism bias and underestimate the time needed.

Different agile frameworks use different metrics, but the commonly-used velocity and cumulatative flow metrics are the typical guardrails for mitigating planning bias. There are certainly other metrics, but I'll talk briefly about velocity because I think it most closely addresses the concern.

How Velocity Mitigates Planning Bias

Velocity is a metric that looks at historical rates of delivery, and uses a moving average or other statistical functions to smooth perturbations. The velocity metric is probabilistic, and while it often works best with a baseline reference class to anchor the metric, in pragmatic usage it's actually less sensitive to that than one might think. The real determinant of how reliable velocity is as a predictive metric is the consistency of the estimation process.

The key to velocity's widely-accepted success in agile planning is a consistent estimation process. It actually matters very little whether the estimation process is optimistic, pessimistic, or ignores the reference class altogether, so long as the estimation process is applied similarly across iterations. Over a sufficient time window, a consistent estimation process will trend towards the sustainable rate of delivery.

Inspect-and-Adapt Caveats

While a consistent estimation process is essential to reliable forecasting, that doesn't mean your project's estimation process can't change or improve. The notable caveat here is that iterative or incremental improvements to the estimation process will create small perturbations in delivery rates, but will generally be small enough to be swallowed by the smoothing function.

As a related caveat, it's also possible to make radical changes (hopefully improvements!) to the estimation process. When that happens, you either need to discard historical data or apply a "fudge factor" to account for the changes in the methodology. This may lead to less forecasting reliability from the metric in the short term, but can often be worthwhile if it leads to higher sustained reliability in meeting forecasts.

1

It's not the estimating together part of planning poker that helps that much (although that can reduce individual biases). It's the velocity tracking part which lets you apply historical data to help with future estimates.

E.g., if in the previous sprint we tried to complete 50 points worth of work, but only finished 40, then our rate is probably closer to 40 than to 50.

Here's another example of using historical data to improve the accuracy of estimates (more complex in terms of data collection; also, it uses probability distributions).

Other methods that I've seen work:

  • Decomposition (breaking down large tasks and estimating the smaller tasks)
  • Training (e.g., calibrating estimators, a method of using training data to improve people's estimating abilities)

Don't forget about the idea of estimate convergence (early estimates are way off, and they'll improve more and more as work progresses).

  • Welcome to PMSE. Due to the danger of link rot, would you consider including the relevant bits of information from the links in your Answer? – Sarov Oct 18 '18 at 14:48
  • Thanks @Sarov. It's a lot of information! I think I can add some more info and remove some of the links, is that reasonable? – Ivan Maeder Oct 18 '18 at 15:14
  • Sure. Also, you don't have to remove the links; just also include key info just in case the links decay. – Sarov Oct 18 '18 at 15:55
  • Answer updated! – Ivan Maeder Oct 18 '18 at 16:13

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.