How can I improve measurement of Sprint Goals that are based on soft skills or communication processes?

Question

Background

I am currently coaching a non-development team on Scrum practices. The team is focused on back-office, administrative processes. This makes the work somewhat of a pull-queue where Kanban would be a better fit if the actual completion of work-product were not externalities to the team; as it is, Scrum was chosen a the framework due to its focus on stand-ups and retrospectives, which the organization believes will improve intra-team communication and lead to incremental process improvement.

"Fail Early" and "Improve Intra-Team Communications" as Sprint Goals

Recently, the team identified "fail early" as the Sprint Goal for an iteration. The idea was that tasks that can't be completed on time may not have alternative solutions or work-arounds---the business model accepts the risk that some percentage of tasks will fail, regardless of process efficiencies---but that these impending failures needs to transparent and visible to the entire team.

The goal of making early failures visible ties directly into the corollary objective of improving intra-team communication. This goes somewhat beyond the task coordination and blocker-identification of a daily stand-up; it requires a bit of a paradigm shift in the way that the team communicates with one another on a routine basis.

To the extent that the goal and its corollary was chosen by the team as a Sprint Goal without a concrete metric to determine if the Sprint Goal was done or not-done, I take responsibility as the Scrum Master for allowing that to happen. I still believe that these were valid objectives for the iteration, but they lack concreteness and a "definition of done." In retrospect, neither of these things has a self-evident performance indicator that can be tracked with any precision.

How to Measure Semi-Subjective Process-Improvement Goals

"Fail early" is somewhat subjective, but I believe I could potentially measure that by calculating the elapsed time from work-item start to its announced failure, and then tracking the "failure lead time" (e.g. the time between when a task is declared "failed" and its original due date). Perhaps there is an even better metric that I haven't considered yet; I'm certainly open to suggestions in that regard.

"Improve intra-team communications" is squishier. Effective communication is a soft-skill. Anything I can think of measuring (e.g. email volume, status fields updated in the job queue, etc.) are at best proxy metrics, and not particularly accurate ones at that. So, other than polling people for how well they feel communication is going, I can't think of a practical way to measure this necessary (but perhaps poorly-framed) objective.

Given the stated Sprint Goals, and assuming a priori that the organization sees value in achieving those two goals:

1. How can I measure them in a concrete, meaningful way?
2. How might I adapt these (or similar) soft-skill goals in the future to better identify valid, trackable metrics?

Answer 1

I think the metric you suggest for "fail early" is probably close to optimal. It might be interesting (although probably too costly) to track the time when the work item was first identified as "in trouble".

1. Time the work item was initiated
2. Time when the work item was first identified as "troubled"
3. Time when the relevant stakeholders identified this work item as "failed".

The elapsed time between 2 & 3 may provide some interesting information for the second metric.

Failure analysis - Since the goal is to fail early, conduct a failure analysis of the work item. If Intra-Team Communications contributed to the failure, score it as a communications fail. Track either the number or percentage of communcations failures.

I'm not sure I understand item #2, but I think what you're asking is how to create meaningful metrics atop soft goals. I think the answer is to look for the impact that the soft goal has on hard business objectives.

Answer 2

For "Fail Early" a valid measure in conjunction with (or in place of) chronological lead time may be the level of effort devoted to the task prior to identifying failure. That will give you a better idea of the resource impacts and whether or not you are saving $$ as well as time. For example, assume I am budgeted 50% effort on a three-week task (so 1.5 weeks of FT effort over that period), and I identify failure after two weeks after spending 100% of my time on the task. Does the benefit of flagging failure one week early outweigh the costs of my spending more time than planned on the task?

For "Improve Intra-Team Communications" I'm not sure if polling is a good idea. Assuming your team is small the data you generate will be sketchy at best. And if meeting sprint goals is tied into the team's bonus structure it would be relatively easy for there to be collusion. Maybe a better approach would be for you to track the frequency of issues raised that seem to be related to poor communications. I've been in any number of situations where a team member has raised issue X and upon investigation it turns out this is because of some kind of communications breakdown.

Answer 3

Improve Intra-Team Communications

Because communications quality is both subjective and highly personal I don't think there's going to be an easy measure for this. I'm also not convinced that it can be marked as 'done' at the end of a two or three week sprint because, as you point out, it's more of a long-term measure.

That said, I think your approach to measuring success is broadly correct. Presumably there was some driving force behind wanting to improve intra-team communications - a lack of face-to-face meetings, too many emails, re-work due to poor comms etc. - and ideally you have some benchmarks to work from. Even without benchmarks you could use a Likert scale poll to determine whether people think comms have improved. More quantitatively I think you could measure whether everyone is attending and contributing in stand-ups; whether documents are being shared on (for example) a wiki or shared drive; how many formal meeting requests were made (as opposed to face-to-face catch ups). It will all depend on how the team currently works and where it wants to get to.

Fail Early

I tend to agree with Mark and Doug's approaches on this but, again, I think it's difficult to measure 'completeness' of this task after only a few weeks - unless your measure of completeness is simply that you have a documented measure for early failure (of the type Doug and Mark have suggested). I actually think the latter is an acceptable outcome but I may have misunderstood what you want to get out of the sprint.

Answer 4

Both of these goals are really bad goals. Neither is measurable because neither really matters in the least. The answer is to ask why either of those two things matter to the team. What is the purpose of improving intra-team communication? Why do they want to fail early? Both sound nifty as buzz-words, but they don't really mean anything in and of themselves. Figure out what they are trying to accomplish by doing these things, and make that the goal (this will more than likely be measurable).

It's like a user story: As a (team member) I want (fail early) so that (here is where the measurable goal lies).

Answer 5

Disclaimer : I am not a Scrum expert, and I do not have any experience with non-dev teams. Thus my answer is both theoretical and imbued with Lean/Kanban thinking

On goal-related metrics and measurability

The goal of a goal is at least triple :

• Command Help the team stay focused and make better choices, thus help management set priorities
• Communicate Help stakeholders understand what the team is doing
• Control Make sure that, in the end, we know if we succeeded or not

Considering this, a goal must be defined so that it is possible to know if it was accomplished or not. In other words a goal should be accompanied by a definition of done.

At this point if you have a metric that everybody understands and is relevant from a business point of view : fine. As this metric is meaningful for your stakeholders, the goal will achieve its triple purpose.

But if you don't have any metric don't bother trying to calculate things that do not provide any interesting information to your stakeholders (like e-mail volume). Instead try to have the team (including stakeholders) answer this question : when will we know that our goal is achieved?

On failure

The definition of "done" must help you get a good understanding of the reasons why the team has failed to achieve its goal. Keep that in mind when defining a metric that should define success/failure conditions, since you will have it as the root of the failure analysis. Your goal here is to facilitate understanding.

On subjective goals

My advise about how to deal with subjective goals is don't try to rationalize them. Let them be subjective. Let's take the "Improve intra-team communications" example. The only good measure here is probably to ask the team members about it. During the Sprint Retrospective Meeting ask the team members (including the stakeholders) if they think that the goal is achieved and why. Ideally you would already have defined the "why" part before the Sprint. A quick debate will then reveal if the goal is achieved or not, and it is likely to provide more information than the result of a weird calculation (At the end of the Sprint "I don't even know what you did!" is better than "Our +communication rate+ is 0.473")

Answer 6

I'm quite certain that is not possible to measure subjective goals and their related actions, because their measurement will be subjective too, but a measurement must be objective.

However, subjective goals usually appear when the team doesn't really know how to tackle a certain problem and starts experimenting. That's why they are usually adviced to measure and observe the outputs (like lead time and throughput), because if they show a positive difference then the counter actions worked, and those outputs can be measured objectively.

What if you count the appearance (mention) of a problem and check how this data looks like week by week? Let's say that the sentence "we need to fail early" or "we should have failed sooner" is said by one of the team members. You count the occasions, or ask the team members to do it, and at the end of the sprint you check this data. If the count is almost zero, means that you kind of solved that problem.

A while ago we weren't sure that we really need continuous delivery. So we counted the occasions when we could have done better if we had continuous delivery. We had 5 occasions in 2 months, so we didn't implement it, because we were good without it.

I think this technique can work for your team as well. If the members less frequently mention that "they have to fail earlier" might mean that they fail early enough for them. Or, if they don't feel the need to mention that the intra-team communication must be improved might mean that it is good enough for them.