How should a Scrum development team track KPIs (key performance indicators) besides sprint commitments?

Question

The most obvious KPI for a development team that uses Scrum is the ability to hit sprint commitments. But is there any other efficient way for such a team to track KPIs?

Answer 1

So this is something I'm dealing with right now in AOL. The constant desire of business to measure seeps even into agile organizations. The key is to make sure you are measuring in a way that does not damage the fabric of agile for the sake of measuring.

We must always be very careful and mindful of Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." Man's ability to game the system is pretty much boundless. Make the measure a target and we will find a way to hit it. Overall success needs to be measured by things they can't directly influence like customer NetPromoter scores.

Now that being said, at the agile team level I still need to find some way to measure the teams (as an agile coach at AOL, I work with over 16 teams). The first thing I did was to setup management expectations of what our metrics are for. Instead of metrics being to set goals, metrics are lagging indicators that tell us a team may need help. Just because a team's velocity has dropped, does not mean they are doing something wrong. It could be outside influences.

I use Jeff Sutherland's three measures of "escaped defect rate", "cycle time" and "happiness index", plus one of my own "Commit to Done".

Escaped Defect Rate: In agile the ultimate measure of success is "Working Software." So monitoring how many bugs escape into the wild, and making the team accountable for this, is one of the first measures of a team.

Cycle Time: The faster you get from "In Progress" to "Done" the more work you can get done overall. I monitor team's cycle time and when it spikes, it tells me something is going on. Right now, with my current teams, it tells me the quality of the product backlog is poor leading to a lot of questions and churn during coding.

Happiness Index: Happy teams are more productive. Anonymous survey tools are one of the best ways to track this. This metric is more a measure against how the company is doing supporting the team.

Commit to Done: Where cycle time is a measure of productivity. I find this to be one of the best ways to promote and measure predictability. If a team commits to 30 stories, and only delivers nine, then the product owner has only a 30% chance of getting something they care about. If a team instead commits to ten and delivers nine, then the product owner has a 90% chance of getting what they wanted. Better predictability leads to more trust and usually to management and product owners interrupting the team as much. When you can consistently deliver what you say you will, you get a lot of lee way.

One caveat- You'll never get to 100% Commit to Done. Any team that does is gaming the system. 80% is probably the highest you'll ever see consistently. This is because as a team finishes work, they will reach into the backlog to pull in new work.

Answer 2

Based on EBMgt (Evidence Based Management Guide)

1. Current Value

Revenue per Employee. Gross Revenue / #employees

Product Cost Ratio. All expenses in the organization that develops, sustains, provides services, markets, sells, and administers the product or system.

Employee Satisfaction. Engaged employees that know how to maintain, sustain and enhance the software systems and products are one of the most significant assets of an organization.

Customer Satisfaction. Sound management, solid software, and creative, fulfilled employees.

2. Time to Market

Release Frequency. The time needed to satisfy the customer with new, competitive products.

Release Stabilization. The impact of poor development practices and underlying design and code base. Stabilization is a drag on competition that grows with time.

Cycle Time. The time (including stabilization) to satisfy a key set of customers or to respond to a market opportunity competitively.

3. Ability to innovate

Installed Version Index. The difficulty customers face installing a new release .The relatively low value of new releases, or even the # of customers that are evaluating alternatives.

Usage Index. Determines a product that is burdensome and difficult to use and excess software that must be sustained even though it is rarely used.

Innovation Rate. Growth of technical debt caused by poorly designed and developed software. Budget is progressively consumed keeping the old software alive.

Defects. Measures increasingly poor quality software, leading to greater resource and budget to maintain it and potential loss of customers.