How to Calculate Effort Variance in Scrum

Question

My team currently works in 3 week sprints and we are capturing Key Performance Indicator (KPI) for each sprint. One of the KPIs is Effort Variance (EV) per sprint

My question is at the end of the sprint, should I consider Effort only for Resolved user stories or for all user stories irrespective of whether they were closed or not. Let me try and explain with an example

Sprint 1 started from 26th Feb 2020 till 20 Mar 2020. At the start of the Sprint we estimated that we will do 10 user stories with an estimated effort of 500 Hours. At the end of the sprint we realised that we were able to complete only 5 user stories which took 200 hours to complete. The effort that was spent on the 10 user stories in the sprint was around 520 Hours.

In this scenario for calculating Effort variance, should I consider only effort spent in completing 5 user stories or should I consider the effort spent for all the 10 user stories?

Answer 1

You can't calculate EV on the 10 items yet because they aren't done, so you don't know what the total effort to complete them is. You could calculate EV on the 5 that were completed if you had estimates for those 5 independent of the other 5.

Answer 2

TL;DR

In this scenario for calculating Effort variance, should I consider only effort spent in completing 5 user stories or should I consider the effort spent for all the 10 user stories?

You should use only the stories completed. You can't meaningfully measure deltas on effort expended on work you haven't completed.

It's worth your while to identify the business or managment objective of measuring "effort variance." I could see the point of measuring actual time/effort spent against estimates if you were doing earned value management, trying to refine your estimation process, or maybe even measure labor costs across multiple Sprints. However, measuring person-hour deltas as a first-class KPI in either a time-boxed or continuous-flow system is generally an anti-pattern.

How to Measure Productivity Incorrectly

"Effort variance" sounds a lot like an anti-pattern where the management team is trying to measure how well (or how poorly) the team is doing in meeting management targets for productivity. In practice, this is usually a proxy metric for productivity when targets are set externally, or when the goal is to "work on lots of work items" rather than achieve specific goals or milestones.

If you are working with time-boxed iterations like Sprints, it's generally more useful to measure successful completion of Sprint Goals than it is to measure (or worse, retcon) historical estimates or effort expended. In time-boxed frameworks, estimate deltas can identify patterns of poor estimation or indicate the presence of hidden process inefficiencies, but while they can feed a continuous improvement process like Sprint Retrospectives they rarely have any long-term tracking value.

Likewise, in continuous-flow systems like Kanban, you generally want to measure things like cycle time, lead time, or Takt time rather than estimation deltas. Continuous flow systems don't use time-boxed iterations or rely on per-item estimation techniques; they use pull queues and statistical models instead. So, tracking the delta of actual values against initial estimates for a very specific time window works against such frameworks.

Solve for X, Not Y, in an X/Y Problem

Once you understand why you're being asked to measure time spent on specific work items, or why you're being asked to measure deltas in your estimates and labor hours, you can solve for those things more directly. If you're being asked to track something, find a KPI that:

1. Measures the actual thing you care about tracking. (Hint: Estimation deltas are never the thing anyone directly cares about.)
2. Can be tuned within your process. Measuring things that can't be modified or adjusted is often a waste of time.
3. Optimizes your outcomes. Unless you can draw a straight line from a metric like time_or_effort_spent / time_or_effort_estimated to a demonstrably improved outcome, the metric adds zero business value.

Getting to "good enough" in the estimating or planning process is part of any good framework's continuous improvement process. However, since estimates don't guarantee outcomes, estimates (or their variances) are rarely useful as first-class project performance indicators.

If You Do It Anyway...

Think of "effort variance" as the variance between your initial estimate of how much effort a work item is expected to take, and then compare that planning value to the effort actually expended to complete that item. For example, if you have 10 Product Backlog Items in a Sprint, each item has an initial estimate (using either time or relative effort) that you can measure against team resources actually expended on that work item. If you're truly measuring effort rather than time or money, you can't track against work not yet completed except in very broad terms. In that case, you really want to measure time directly.

Think of it this way. Given the same 10 work items, if you say:

We completed half our work items. We spent 80% of our time on the 50% that got done, and the other 20% on the other 50% that didn't.

what are you actually learning from the metric? You probably want to say one or more of the following things instead:

• We estimated that we could complete 10 items this Sprint, but our actual capacity was only 50% of the estimate.
• These five items took 200% (or possibly more, depending on the initial estimates) of planned effort/time.
• The team overcommitted by 50% this Sprint.
• The five items completed took 80% of the team's capacity this Sprint. The remaining 20% was insufficient to complete any of the work items forecast for this iteration.

To put it another way, you can either report effort expended on buckets such as "done/not-done" or on specific completed items to get a useful variance. Time/effort spent on incomplete items can be useful for determining muda, but it can't accurately measure how long (or how much effort) it would have taken to complete it successfully.

Don't...

• Don't measure incomplete work piecemeal.
• Don't measure deltas in man-hours if you're estimating in story points.
• Don't treat estimates as money-back guarantees.

Do...

• Do measure completion of Sprint Goals, project goals, or milestones.
• Do measure significant deltas in estimates vs. actual progress using burn-downs (for early detection) or post-facto for retrospectives.
• Do treat estimates as forecasts to be refined.
• Do treat the estimation process as an opportunity for continuous improvement.

Measuring how much you ran over completing a subset of user stories may hold value if it allows you to correct your estimation process, or to adjust scope dynamically within a Sprint if you detect a significant variance early enough. Measuring how much time you sank into incomplete stories after the time box has expired probably won't tell you anything useful, other than how much time/money to treat as a sunk cost for that Sprint. Stick to the former; the latter is about as non-agile as it gets.