Agile ("Scrum") "Metrics" Question

Question

My team has recently transitioned from essentially zero project management (other than the "put out fires as they happen while trying to stay on top of current workload") to something more Agile with roots/tentacles into Scrum.

The situation has improved greatly, but our manager is asking for a roadmap/how to know that things are going better. I don't think he's disputing or questioning whether or not our project management has improved; rather, I think he's trying to figure out how much the situation has improved and how to quantify this. To me, this is a difficult question to answer because a) I've only been here two years, and b) it's difficult to quantify improvement to ~0 to something.

Any thoughts? I saw this question, but it didn't quite get to exactly what I was looking for.

Answer 1

Basically, you do not have any data at the moment that you can use to quantify how much you've improved. So start collecting it immediately. Be it velocity, lead times, cycle time or qualitative data via questionnaire the main thing is that you start collecting it so you have some kind of historical data which you can use in the future.

What I would do in your situation is to create a questionnaire for the team and all of the relevant stakeholder groups the team is interacting with. Ask questions like "how much has our predictability improved in X months" and give a scale of -5 to 5 or something like that. Also ask opinions on the current situation so you get some kind of baseline, for example, "how satisfied are you with the team's performance" etc.

Hopefully, enough people answer your questionnaire and you get some data to present to your manager.

Also, there's no roadmap for a change initiative. Or not a reliable one, at least.

Answer 2

Have lead times improved?

See if you can find some historical information that shows when requirements were first raised and when they made it in to production. If your project management process has improved it could well have resulted in the lead times dropping.

Have the number of production incidents improved?

Most organisations retain some information about production incidents. You might have to plough through some old emails, but you have a good chance of working out if the code in production is now more reliable.

Do business people in your organisation think things have got better?

Survey the business people in your organisation. Ask them questions like: "Do you have more or less confidence in your development team now compared to 1 year ago?"

Answer 3

When you started the changes, what did you expect to change? Initially, you do not have roadmap or planning, as I understand. Try to think in several directions:

 1) what changed for you - which improvements helped you?
 2) what changed for team - which improvements helped your team?

As a second step, when you decide what changed, you could try to understand how to measure this. Whether through some key performance indicators or metrics or through feedback from the team or other stakeholders.

Answer 4

Apart from measuring number of incidents, lead times, etc. I would also focus on how the work culture is changing and try to quantify that, because it is very important to keep everyone happy and so that they can work to the best of their ability.