1

My team currently works in 3 week sprints and we are capturing Key Performance Indicator (KPI) for each sprint. One of the KPIs is Effort Variance (EV) per sprint

My question is at the end of the sprint, should I consider Effort only for Resolved user stories or for all user stories irrespective of whether they were closed or not. Let me try and explain with an example

Sprint 1 started from 26th Feb 2020 till 20 Mar 2020. At the start of the Sprint we estimated that we will do 10 user stories with an estimated effort of 500 Hours. At the end of the sprint we realised that we were able to complete only 5 user stories which took 200 hours to complete. The effort that was spent on the 10 user stories in the sprint was around 520 Hours.

In this scenario for calculating Effort variance, should I consider only effort spent in completing 5 user stories or should I consider the effort spent for all the 10 user stories?

6
  • 2
    What will you do with this metric? How will it help your team meet its Sprint Goal, or the Definition of Done?
    – Todd A. Jacobs
    Mar 23, 2020 at 11:58
  • "Notwithstanding 'sprint' terminology," I think that the most-significant finding here is that only 5 stories burned-up all the time, while another two hours each was spent (who cares) on the other two. I'd say the team completely failed to predict how much time would actually be taken on the five, and that it probably didn't spend enough effort on the other five to know whether they also had been under-estimated. ("Of course, likely.") The team's initial projections turned out to be "tremendously wrong," and that's the lesson to be learned here, however you present it in metrics form. Mar 23, 2020 at 14:34
  • @ToddA.Jacobs The team already knows that they were not able to complete 50% of the user stories and the effort that was spent. The KPI metrics is more for senior management who would like this data to be captured on a sprint on sprint basis. I did check with the quality department and they have instructed me to use the effort spent on the entire 10 stories (irrespective they are finished or not) vs the planned effort for the 10 stories to calculate EV
    – Manuj
    Mar 24, 2020 at 4:32
  • Related: pm.stackexchange.com/a/20646/4271
    – Todd A. Jacobs
    Mar 24, 2020 at 21:16
  • Effort variance is simply (Actual Effort - Planned Effort) / Planned Effort * 100. But honestly, saying you are at 4% variance with only 50% of your planned work completed seems nonsensical, so please don't do that. Find a different way to report on the problem.
    – Todd A. Jacobs
    Mar 24, 2020 at 23:01

2 Answers 2

2

You can't calculate EV on the 10 items yet because they aren't done, so you don't know what the total effort to complete them is. You could calculate EV on the 5 that were completed if you had estimates for those 5 independent of the other 5.

3
  • Probably what I would do is to act as if the sprint effectively consisted of five items, not ten. Had the team chosen to do only these five, they would have finished them and felt "successful." Effectively, they erred by biting off more than they could chew, and by failing to realize this at the time. On the one hand, the sprint did complete 50% of the work. On the other hand, the team over-estimated the sprint's ability to accomplish work by 200%. "Why did this happen?" That'll be what everyone wants to know. Mar 23, 2020 at 14:39
  • @MikeRobinson It's a red herring. If you can meet the Sprint Goal with only 50% of the planned work completed, why was the rest of the work even on the Sprint Backlog? Alternatively, if the Sprint Goal can't be met less than halfway through the Sprint, continuing the death march makes no sense. "Why did this happen?" is likely due to the lack of a coherent Sprint Goal, rather than a failure to track variances per se.
    – Todd A. Jacobs
    Mar 24, 2020 at 23:06
  • +1 for an accurate (and succinct) answer. I'm answering for X in the X/Y, but I think your answer addresses the OP's Y extremely well.
    – Todd A. Jacobs
    Mar 24, 2020 at 23:09
2

TL;DR

In this scenario for calculating Effort variance, should I consider only effort spent in completing 5 user stories or should I consider the effort spent for all the 10 user stories?

You should use only the stories completed. You can't meaningfully measure deltas on effort expended on work you haven't completed.

It's worth your while to identify the business or managment objective of measuring "effort variance." I could see the point of measuring actual time/effort spent against estimates if you were doing earned value management, trying to refine your estimation process, or maybe even measure labor costs across multiple Sprints. However, measuring person-hour deltas as a first-class KPI in either a time-boxed or continuous-flow system is generally an anti-pattern.

How to Measure Productivity Incorrectly

"Effort variance" sounds a lot like an anti-pattern where the management team is trying to measure how well (or how poorly) the team is doing in meeting management targets for productivity. In practice, this is usually a proxy metric for productivity when targets are set externally, or when the goal is to "work on lots of work items" rather than achieve specific goals or milestones.

If you are working with time-boxed iterations like Sprints, it's generally more useful to measure successful completion of Sprint Goals than it is to measure (or worse, retcon) historical estimates or effort expended. In time-boxed frameworks, estimate deltas can identify patterns of poor estimation or indicate the presence of hidden process inefficiencies, but while they can feed a continuous improvement process like Sprint Retrospectives they rarely have any long-term tracking value.

Likewise, in continuous-flow systems like Kanban, you generally want to measure things like cycle time, lead time, or Takt time rather than estimation deltas. Continuous flow systems don't use time-boxed iterations or rely on per-item estimation techniques; they use pull queues and statistical models instead. So, tracking the delta of actual values against initial estimates for a very specific time window works against such frameworks.

Solve for X, Not Y, in an X/Y Problem

Once you understand why you're being asked to measure time spent on specific work items, or why you're being asked to measure deltas in your estimates and labor hours, you can solve for those things more directly. If you're being asked to track something, find a KPI that:

  1. Measures the actual thing you care about tracking. (Hint: Estimation deltas are never the thing anyone directly cares about.)
  2. Can be tuned within your process. Measuring things that can't be modified or adjusted is often a waste of time.
  3. Optimizes your outcomes. Unless you can draw a straight line from a metric like time_or_effort_spent / time_or_effort_estimated to a demonstrably improved outcome, the metric adds zero business value.

Getting to "good enough" in the estimating or planning process is part of any good framework's continuous improvement process. However, since estimates don't guarantee outcomes, estimates (or their variances) are rarely useful as first-class project performance indicators.

If You Do It Anyway...

Think of "effort variance" as the variance between your initial estimate of how much effort a work item is expected to take, and then compare that planning value to the effort actually expended to complete that item. For example, if you have 10 Product Backlog Items in a Sprint, each item has an initial estimate (using either time or relative effort) that you can measure against team resources actually expended on that work item. If you're truly measuring effort rather than time or money, you can't track against work not yet completed except in very broad terms. In that case, you really want to measure time directly.

Think of it this way. Given the same 10 work items, if you say:

We completed half our work items. We spent 80% of our time on the 50% that got done, and the other 20% on the other 50% that didn't.

what are you actually learning from the metric? You probably want to say one or more of the following things instead:

  • We estimated that we could complete 10 items this Sprint, but our actual capacity was only 50% of the estimate.
  • These five items took 200% (or possibly more, depending on the initial estimates) of planned effort/time.
  • The team overcommitted by 50% this Sprint.
  • The five items completed took 80% of the team's capacity this Sprint. The remaining 20% was insufficient to complete any of the work items forecast for this iteration.

To put it another way, you can either report effort expended on buckets such as "done/not-done" or on specific completed items to get a useful variance. Time/effort spent on incomplete items can be useful for determining muda, but it can't accurately measure how long (or how much effort) it would have taken to complete it successfully.

Don't...

  • Don't measure incomplete work piecemeal.
  • Don't measure deltas in man-hours if you're estimating in story points.
  • Don't treat estimates as money-back guarantees.

Do...

  • Do measure completion of Sprint Goals, project goals, or milestones.
  • Do measure significant deltas in estimates vs. actual progress using burn-downs (for early detection) or post-facto for retrospectives.
  • Do treat estimates as forecasts to be refined.
  • Do treat the estimation process as an opportunity for continuous improvement.

Measuring how much you ran over completing a subset of user stories may hold value if it allows you to correct your estimation process, or to adjust scope dynamically within a Sprint if you detect a significant variance early enough. Measuring how much time you sank into incomplete stories after the time box has expired probably won't tell you anything useful, other than how much time/money to treat as a sunk cost for that Sprint. Stick to the former; the latter is about as non-agile as it gets.

1
  • For all intents and purposes, this team completed 5 tasks and ignored the other 5. The root cause management problem is that the team apparently wildly-overestimated what it could achieve ... and I think it's very, very important to show this properly. Within the sprint, the teams did complete the five, and, had they only agreed to do five, they'd call this sprint a success. "Average" is really going to be misleading here because the "standard deviation" is off the scale. The team, probably realizing they'd bit off too much, did land five of the ten airplanes. Mar 25, 2020 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.