Is a zigzag, sawtooth velocity a problem if the trend is consistent?

I am supporting and coaching several scrum teams working on different areas of the same software project. Sprints are 4 weeks long, except that sprints spanning the Christmas/New Year holidays are 5 weeks long. (The first six or so sprints were 3 weeks, before the teams decided to change them to 4.) All of the teams' velocities have a very noticeable pattern of zigzagging, where velocity alternates between higher and lower values every single sprint. In the graphs below, the Y axis is points Done per sprint; the gray lines are 10 story points. Each team has an independent backlog and estimates its own stories, so while points are not consistent between teams they are consistent within teams. In raw numbers one team completes twice the points of the other teams, so the graphs are normalized across teams under the assumption that each team's average velocity is equivalent, to prevent "point competition".

The teams explain this variation as having mostly Done stories carry over to the next sprint, so that some sprints finish fewer stories and other sprints finish more. Teams typically commit to a third or a half more points than they actually finish, and many of these extra points are "In Development" at the end of each sprint. Developers are resistant to smaller stories or shorter sprints as they "wouldn't have time to finish anything" and "would always be in Scrum meetings".

This is related to Team velocity fluctuates a lot, how to find the root cause?, but I think it is a much more specific issue. From those answers, I considered impediments and unbalanced work as possible explanations, but they aren't enough by themselves to cause this extreme zigzagging. Turnover on the teams is low and much of the velocity variation has occurred during periods of stability.

1) Have you seen such consistent zigzag patterns in velocity on your teams?

2) Does this pattern suggest a problem?

3) If so, what is the problem, how it is harmful, and how is it correctable?

• Welcome to PMSE! How many story points are "not done" at the end of each of these sprints? And what's the scale on your Y-axis? Jan 5, 2015 at 23:36
• What is the Y-axis of these graphs? How much variation are we talking about here? Is this +/- 2 or 3 story points or 20? Jan 6, 2015 at 18:55
• Each line is 10 points, so velocity regularly swings by 20 to 30 points between sprints.
– Joel
Jan 7, 2015 at 18:05
• The upward trend is more concerning than the zigzag. Typically that indicates everyone is padding their estimates more and more. Try comparing this with an actual # of items delivered chart as well. Jan 7, 2015 at 20:14
• If, at the end of a sprint, you have a 5 SP story with 1/2 SP of effort left on it, for how many SP does this story get planned in the next sprint? Jan 12, 2015 at 16:02

The zig-zag is usually caused when a majority of stories are longer than the length of the sprint, so you end up with one sprint where little is completed, and one where lots of stories are finished and velocity is high. Plot yourself a histogram of stories against how long they take to complete (actually take, not the story points), and you'll probably find this is the case. This also makes compelling evidence to persuade people to fix it.

Even if the overall trend is upwards, at the moment you're getting feedback on a large number of items which have been in play for two sprints or possibly even longer. That becomes expensive to fix when the feedback tells you you got it wrong, since the knowledge involved in the work decays over time, and further work cements it in place. The whole point of Scrum and Agile methodologies is to be able to change course if something is wrong.

So, just because the velocity is healthy, doesn't mean that this isn't going to cause an issue. Check your bug count and whether that's rising, and find out how long it takes to fix those bugs.

You have a couple of choices for smoothing the graph:

• Lengthen the sprints. I actually did this for one team because we were on one-week sprints and the overhead of performing all the rituals every week was too much anyway. In general, though, I prefer to...

• Slice the stories more finely. Remember that the purpose of the showcase is to get feedback from and establish trust with stakeholders. If you can focus on that, you can probably slice the stories up more thinly. If you can get one scenario working, or a bit more of the output or input, or another business rule, then you have something on which you can get feedback.

If you encourage the development team to manage this split themselves, and help them focus on the need for feedback, they will be able to see that the overhead isn't huge; it's the same discussion, just sliced up differently.

I disagree with Jamezrp's assertion that mature devs get better at this. We get better at refusing to put estimates on things we've never done before and insist on doing spikes instead, and we get better at predicting managers' desire to have accurate estimates and add more points as buffers if they insist, making it appear that we have better predictability (and higher velocity). This gaming is extremely common in Scrum teams where pressure for accurate estimates is applied. If you track the length of time it takes to deliver whole features, and do it by measuring in time rather than points, this will give you a more accurate measure of true progress and show you whether the rate of delivery is really going up or down.

A truly mature team slices things thinly enough that the points don't really matter any more, and you can usually assign an "average" number of points to a story with little impact.

And the most successful teams of all are those that focus on feedback over accurate estimates (an oxymoron if ever there was one, hence the #noestimates movement).

In theory and according to scrum, it is a problem. The amount of variation points to the a misunderstanding of the work, and if it's consistently off, then the teams determining how many points a given sprint is are doing it wrong. That means they don't know either how to determine points or what the work they're actually doing entails.

In practice, it's just fine. In fact I'd say you have a fairly healthy stability because it tends to be roughly the same. With 4-week sprints (I have my team work on one-week sprints with 1-month planning sessions) you are going to get higher variance unless the team knows exactly what they're doing completely in advance and runs into no hiccups along the way. I've never seen that happen, no matter the maturity level of the teams...if someone is sick, you'll see a variance. Only robots would always have the same numbers regularly.

No, looking at your charts, it looks healthy though you may want to see if any individual stories are getting over/underestimated regularly. I've seen some teams mark investigative stories highly only to have them done in zero points time, and I've seen "easy" zero point stories turn into weeklong hauls. The more mature the team, the lower the variance, and with a little digging you'll be able to see where it's coming from.

There are specific rules in diagnosing a run or control chart. I quickly found this link that has a high-level explanation:

http://www.skymark.com/resources/tools/control_charts.asp

Everything produces variance, including an automated or robotic process. You need to determine whether the variance is "common cause", i.e., purely random, statistical noise; or "special cause," i.e., something is wrong with the process. The way a variance is produced will indicate with some degree of validity (meaning there are both type I and II errors in doing this type of analysis) whether or not the variance you are noticing is random or not.

Depending on the source, you'll see different "indicator" rules but some common ones are:

1. six or seven consistent results above or below the mean
2. constant up and down results exceeding 14 observations (Team 1 is exhibiting this between observation 8 to 22 or so.)
3. four or five results above or below one sigma
4. two or three results above or below two sigma (Depending on the y-axis and the calculation of sigma, team 3 might be showing something.)
5. one result above or below three sigma (considered as upper and lower control limits.)

Type 1 and 2 errors are an issue and you need to understand how they occur. If your chart is too sensitive, you could get a ton of false positives and vice versa if too insensitive. So you need to consider your findings as probabilistic, not deterministic.

DISCLAIMER: I don't know a thing about SCRUM so control charts may not be applicable, but I am not sure why it wouldn't be.

1) Have you seen such consistent zigzag patterns in velocity on your teams?

Yes this is a common observation, especially with teams that aren't accountable to a customer or 3rd party for delivery of an increment at iteration end.

2) Does this pattern suggest a problem?

It depends. If what they commit to delivering at the end of the iteration also goes to production or must be delivered to an external stakeholder this is definitely a problem. If your scrum teams have delivery timelines outside of the iteration it may not be a problem for the customer, but should still be investigated as large variances are an indicator of inefficiencies/ineffectiveness on the team.

3) If so, what is the problem, how it is harmful, and how is it correctable?

There are many root causes of this problem. Some common ones are (and please determine if any of these may apply to your situation): stories poorly defined at iteration start, stories not appropriately sized or decomposed, Scope changes on stories after planning, non-visible work is frequently injected into the iteration, large cycle times on stories (this can have several root causes as well), no incentive to honor commitments at the iteration level.

Ultimately this is harmful if your customer frequently doesn't get the value they were promised at the end of the iteration. It can erode trust and confidence in using Agile-Scrum as a delivery framework since mature teams should be able to deliver the majority of their commitments fairly consistently at iteration end.

How to correct the problem depends on the root cause(s). Some corrections include: reducing the duration of the iteration, improving backlog quality and understanding during planning activities, allowing the team to under-commit and then back-filling extra work near iteration end until the team matures its estimation capabilities and commitment confidence, ensuring all real work is tracked, ensuring there is a real incentive at iteration end to meet commitments, introducing work in progress limits to focus the team on cycle times.