8

The most obvious KPI for a development team that uses Scrum is the ability to hit sprint commitments. But is there any other efficient way for such a team to track KPIs?

  • Do you want to have KPIs to measure the team against it's own history, or to compare it with other teams? – nvoigt Mar 8 '16 at 7:03
  • 1
    What is your goal? Using metrics to drive goals is a slippery slope. Godard's Law in full effect. – Joel Bancroft-Connors Mar 8 '16 at 7:09
  • @nvoigt we'd like to measure the team against our own history. – Sebastian Mar 8 '16 at 8:04
  • @JoelBancroft-Connors I'm not sure I'm following. How would such a goal look like and why is it ricky to use metrics to drive it? – Sebastian Mar 8 '16 at 8:05
  • Went ahead and gave an answer with more details. – Joel Bancroft-Connors Mar 8 '16 at 8:10
13

So this is something I'm dealing with right now in AOL. The constant desire of business to measure seeps even into agile organizations. The key is to make sure you are measuring in a way that does not damage the fabric of agile for the sake of measuring.

We must always be very careful and mindful of Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." Man's ability to game the system is pretty much boundless. Make the measure a target and we will find a way to hit it. Overall success needs to be measured by things they can't directly influence like customer NetPromoter scores.

Now that being said, at the agile team level I still need to find some way to measure the teams (as an agile coach at AOL, I work with over 16 teams). The first thing I did was to setup management expectations of what our metrics are for. Instead of metrics being to set goals, metrics are lagging indicators that tell us a team may need help. Just because a team's velocity has dropped, does not mean they are doing something wrong. It could be outside influences.

I use Jeff Sutherland's three measures of "escaped defect rate", "cycle time" and "happiness index", plus one of my own "Commit to Done".

Escaped Defect Rate: In agile the ultimate measure of success is "Working Software." So monitoring how many bugs escape into the wild, and making the team accountable for this, is one of the first measures of a team.

Cycle Time: The faster you get from "In Progress" to "Done" the more work you can get done overall. I monitor team's cycle time and when it spikes, it tells me something is going on. Right now, with my current teams, it tells me the quality of the product backlog is poor leading to a lot of questions and churn during coding.

Happiness Index: Happy teams are more productive. Anonymous survey tools are one of the best ways to track this. This metric is more a measure against how the company is doing supporting the team.

Commit to Done: Where cycle time is a measure of productivity. I find this to be one of the best ways to promote and measure predictability. If a team commits to 30 stories, and only delivers nine, then the product owner has only a 30% chance of getting something they care about. If a team instead commits to ten and delivers nine, then the product owner has a 90% chance of getting what they wanted. Better predictability leads to more trust and usually to management and product owners interrupting the team as much. When you can consistently deliver what you say you will, you get a lot of lee way.

One caveat- You'll never get to 100% Commit to Done. Any team that does is gaming the system. 80% is probably the highest you'll ever see consistently. This is because as a team finishes work, they will reach into the backlog to pull in new work.

  • 1
    "If a team instead commits to nine and delivers ten, then the product owner has a 90% chance of getting what they wanted." Did you mean to say that? Or did you mean: "If a team instead commits to ten and delivers nine, then the product owner has a 90% chance of getting what they wanted.". Feel free to ignore my comment if I am misunderstanding what you were trying to say. – Barnaby Golden Mar 8 '16 at 13:03
  • Great post, +1. I would add "Flexibility" (typically self-survey) which encompasses knowledge- and skill-decentralization, domain expertise and skill set growth, etc. And as blasphemous as it sounds, as time goes on, I find myself less and less buying into a hard defect rate KPI in most orgs. I feel that it's an artifact from dev-driven agile and is not product- or value-minded enough for "the post-Scrum agile era". I've had many teams (laser-focused on value) choose to release 5 updates with edge case/minor bugs vs. 2 of them perfectly done; I guess as with all agile, "context is everything"? – Jeff Lindsey Mar 9 '16 at 15:10
  • @Barnaby- Yes, good catch. Fixed now. – Joel Bancroft-Connors Mar 10 '16 at 8:06
  • @Jeff - I think what you are saying is "Having engineers who can do more than one thing is important."? If so, yes, I agree totally. I coach my teams in simple economics. "In the next ten years there will be more and more call for Full Stack developers. Learn now so you are not left behind." – Joel Bancroft-Connors Mar 10 '16 at 8:07
  • @JoelBancroft-Connors Yep, both in terms of languages and frameworks as well as the actual code base. My current teams do this by simply rating their "comfort levels" with each major area or feature of the product, tagging upcoming work that is an opportunity for learning, and tracking this over time. – Jeff Lindsey Mar 10 '16 at 20:34
0

Based on EBMgt (Evidence Based Management Guide)

  1. Current Value

Revenue per Employee. Gross Revenue / #employees

Product Cost Ratio. All expenses in the organization that develops, sustains, provides services, markets, sells, and administers the product or system.

Employee Satisfaction. Engaged employees that know how to maintain, sustain and enhance the software systems and products are one of the most significant assets of an organization.

Customer Satisfaction. Sound management, solid software, and creative, fulfilled employees.

  1. Time to Market

Release Frequency. The time needed to satisfy the customer with new, competitive products.

Release Stabilization. The impact of poor development practices and underlying design and code base. Stabilization is a drag on competition that grows with time.

Cycle Time. The time (including stabilization) to satisfy a key set of customers or to respond to a market opportunity competitively.

  1. Ability to innovate

Installed Version Index. The difficulty customers face installing a new release .The relatively low value of new releases, or even the # of customers that are evaluating alternatives.

Usage Index. Determines a product that is burdensome and difficult to use and excess software that must be sustained even though it is rarely used.

Innovation Rate. Growth of technical debt caused by poorly designed and developed software. Budget is progressively consumed keeping the old software alive.

Defects. Measures increasingly poor quality software, leading to greater resource and budget to maintain it and potential loss of customers.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.