Should management review the scrum metrics of scrum teams?

Question

I read somewhere that team's velocity, burn-up chart, and other scrum indicators are only for team's internal purpose.

Should anyone outside the team i.e. from management review the team's chart and analyze why they failed/succeeded in the last sprint?

What could be pros and cons of such management review of scrum metrics by management?

Answer 1

Scrum favours transparency. The Scrum Guide points out that the defined Scrum artifacts (product backlog, sprint backlog and the results of each iteration) should be transparent.

However, burn-up charts and velocity are not a part of the Scrum definition and so are not necessarily regarded as artifacts requiring transparency.

As a Scrum Master, my approach is to only make this kind of artifact transparent if there is a clear understanding of what the information means and how it is to be used.

For example, if a manager wanted to see a team's velocity I would first explain to them that velocity is purely a measure of capacity and is not a performance measure. If the manager understood and was happy to view the velocity in that context then I would be happy to make it available. The same goes for burn-up charts, etc.

Answer 2

Burn-up, burn-down, backlog and the other Agile tools are called information radiators, they are very important to minimize project reporting. They allow whoever passes by to know the status of the project, this technique helps in many things; to show the progress of the project, to show the achievements of the team, to make the team committed to the results (as they are self managed), and so many other benefits.

This can never be considered as a disadvantage, as the team will have a recognition after they achieve their goals.

Answer 3

Hmmmm...I appreciate some points brought up by the other answers; others I'm not so sure about. Quotes are mainly from the Scrum Guide as of this writing (any quote from other sources will be noted).

Three pillars uphold every implementation of empirical process control: transparency, inspection, and adaptation.

Transparency means being able to see something (making the implicit explicit), inspecting what is seen, and adapting. The thermostat concept is used a lot in Scrum training.

You set your thermostat to a temperature. The thermostat has visibility into the temperature of the room (metric and feedback). For heating, when the temperature goes below the mark by a certain amount, the countermeasure of turning on the furnace kicks in. When the temperature goes above the mark by a certain amount, the countermeasures are stopped.

What if we covered the thermostat and didn't let it sense the temperature in the room? (Probably gonna wonder why it's not getting warmer in your house.)

Therefore, I tend to err on the side of transparency for everything. And I mean everything. Then, as a coach (no matter the formal role I have - even developers can say, “That’s really not what those data measure.”), I coach and train everyone involved on what's going on. What are the data being collected? Why that data? What does the data mean? What do the data indicate?

At any point in time, the total work remaining to reach a goal can be summed. The Product Owner tracks this total work remaining at least every Sprint Review. The Product Owner compares this amount with work remaining at previous Sprint Reviews to assess progress toward completing projected work by the desired time for the goal. This information is made transparent to all stakeholders.

Various projective practices upon trending have been used to forecast progress, like burn-downs, burn-ups, or cumulative flows. These have proven useful. However, these do not replace the importance of empiricism. In complex environments, what will happen is unknown. Only what has already happened may be used for forward-looking decision-making.

And, the endnote:

Scrum is free and offered in this Guide. Scrum’s roles, events, artifacts, and rules are immutable and although implementing only parts of Scrum is possible, the result is not Scrum. Scrum exists only in its entirety and functions well as a container for other techniques, methodologies, and practices.

Velocity as it is typically practiced is not part of the definition of Scrum - it is an add-on we use to accomplish some form of inspection and adaptation. The definition of total work remaining is something agreed to by the Scrum Team (and the organization as a whole).

Could be a measure of the total number of feature requests (user stories - also not part of Scrum).

Some teams and organizations do not feel that the feature request count is detailed enough; therefore, they turn to story points (not part of Scrum) for more granular detail in measuring work remaining.

All that to say, Scrum, as defined, does not have metrics. Organizations and teams have a desire for data to inspect and adapt. If we're looking at the "wrong" data for the "wrong" reasons, bad things happen. Hence the education and explanation piece being important.

A trainer and consultant I know actually put together a wonderful dashboard to help one of his clients. There are data received from external stakeholders - about the state of the product increment (the only measure of progress - users using something). There are data received from, and which matter primarily to, internal stakeholders. There are data received from individual teams (if you're a multi-team organization), which may matter to internal or external stakeholders...this is usually velocity, burn-ups, and so-on.

If that last one is the only one the organization really focuses on when measuring the success of their organization then I would say they might not be embracing empiricism and transparency.

• External feedback measures may include: Product revenue generation versus cost (return on investment) and customer satisfaction. (The teams might find this information useful and motivating as well.)
• Internal feedback measures may include: Project cost per iteration, time to market, the ratio of new features versus repair and maintenance per iteration.
• Team-level feedback measures: Last, and certainly least, progress - scope, defects, ideal, velocity, and velocity trend.

Is a team (or organization) that goes really fast to deliver a product increment users don't like (customer satisfaction) and aren't willing to pay for (ROI) successful? (Does "velocity" and "burn-ups" measure those things? No. No they don't.)

Is a team (or organization) who burns through money like it's water at a marathon, while taking forever to create a potentially shippable increment, that has no new features successful? (Does "velocity" and "burn-ups" measure those things? Kind of, maybe.)

Is a team (or organization) who completes their forecasted work every Sprint to the level of their definition of done successful? (Does "velocity" and "burn-ups" measure those things? Absolutely. And I have no idea if the team and organization are successful, because we're not measuring - or making transparent - the data necessary to get the whole picture.)

Hope that helps.

Answer 4

Like many things in the world of project management; it depends.

The organizations implementing Scrum (or a variation of Scrum which they have adapted) are numerous and all have different hierarchies, aims, reporting chains and personalities. I am guessing you really mean

Should management review my Scrum team's metrics?

Only you can answer that and to help you it is worth mapping where your team sits within the organisation.

1. Are you working for a Government company that must demonstrate value to the tax payer? (if so metrics are likely to be audited)
2. Are you working for a public-listed company that must demonstrate value to the shareholder and display fiduciary responsibility? (as above)
3. Are you working on a project that sits within a Programme that sits within a Portfolio? (It is likely the Portfolio Manager must report the metrics in aggregrate to Directors)
4. Is your project funded from a centralised budget that must compete with other projects for continued funding? (Metrics then become a yardstick for further funding and are likley to be gamed)
5. Does your project report to a Steering Committee? (Metrics will be a quick fire way of showing progress to dozens of customers)
6. Does the Sponsor understand Scrum and business value or do they come from a different background? (Accountants promoted to Directors like metrics)
7. Is your project liable for audit from internal or external sources? (Metrics will be requested)
8. Does your project fall under PMO for any kind of governance or oversight? (Metrics are one pillar of governance)
9. Does your company have Agile Coaches who provide support and advise on feature team maturity? (Metrics are used as a temperature check across the company of Agile maturity)
10. Do you, or the team, or the department have a track record of failure with low business value? (Metrics are an early warning system)
11. Do you work in manufacturing? (Metrics are critical to production flow)

I can list dozens more considerations that help determine a mature response to whether your metrics should be reviewed by management.

What if I have concerns?

Ultimately, we advocate that the only true metric for a Scrum team is working software. However, that sentiment is an aspiration and is not a mature response to the organisation funding your project which must be balanced alongside dozens or potentially hundreds of other initiatives competing for the budget allocated to your Scrum Team.

You should really consider, if you are uneasy, why you are uneasy.

• Do you fear transparency?
• Do you feel the metrics are sub-par?
• Do you feel management will gamify the metrics or force teams to compete?
• Are the metrics fabricated?
• Are your metrics continually well they would have been higher but X, Y or Z happened every Sprint

I don't fear delivering metrics to any level of the organisation because it gives me a chance to evangelize the Agile values and re-focus management to concentrate on the metrics that truly matter and explain why they matter. Your job is to advocate the system you are implementing.

Should Management provide advice?

Possibly. Probably not but possibly.

Scrum Masters can become blinded to the fact that Scrum is only one framework of many and Scrum has several drawbacks that do not make it suitable for every project.

Management might be able to refer you to previous projects that had success, or they might query things that were not picked up in the Retrospective (why did you over-commit to us in this Sprint, we didn't pressure you) or they may be able to remove institutional blockers or support the hiring and firing of resource since Scrum does not make mediocre team members suddenly effective.

The key is to use the managers effectively as the next level of blockage removal.

What metrics should they be interested in?

Again, this is subjective but the bare minimum should

• Predictability (this shows the team can estimate effectively, don't over commit and can demonstrate regular progress)
• Velocity and Burn Up (this demonstrates the team can give a ballpark figure for completion since we are not funding Feature Teams indefinitely)
• Escaped defects to Production (this demonstrates the quality of the deliverable

All in all, I have a list of around 70 metrics (from code coverage to cycle times) I can track for a Feature Team but management should only be interested in a few that give stakeholder confidence.

Never forget that Agile does not mean don't plan, plans are crucial, it simply means respond to change on a regular basis due to empirical information. Scrum timeboxes allow us to do that but metrics and working software are two sides of the same coin that allow us to demonstrate confidence to the people ultimately paying the team's salary.

Your challenge as an Agile Coach, Scrum Master or insert term here is to manage the relationship pipeline with those managers and let your development team manage the code pipeline.

Are there any counterexamples of Agile teams working to strict deadlines and being tracked mercilessly?

Sure. The entire Toyota Production System which underpins many of the systems we consider Agile is based on exceptionally detailed metrics tracking. Kanban lives and dies by it's metrics. The Tesla and SpaceX teams, lauded by the Agile community, work to exceptionally strict 100 day product roadmaps with detailed metrics provided to management.