How to cope with the Hawthorne effect and sprint improvements?

Question

Hawthorne effect: the phenomenon in which subjects in behavioral studies change their performance in response to being observed.

Background: I'm using Kanban as a tool to analyse and improve the team's internal process. After each sprint (2 weeks) the team suggests and implements improvements, some of them based on the data gathered with the cumulative flow chart.

Given that, what is a good time frame to analyse if an improvement made during a sprint really helps the team, without overlapping with the observed results of other improvements? What are the variables involved that can make the effect last longer or shorter? (project type, team structure, sprint duration, etc.) Should I try to make it last longer or it can damage the production with fake efficient processes in the long run?

Answer 1

This simple answer is, you can't know. When you implement a change and observe a result, it can be due to the change, be a false change like due to the Hawthorne Effect, your bias meaning you see a change but there is none, or be a random result. The only way to know, to arrive at a deduced conclusion that stands up to scrutiny, is to conduct many observations in a controlled environment and with some statistical analysis to rule out randomness and experimenters' bias. And when I say many, I mean many. Not like two or three sprints. Remember, you can flip a fair coin ten times and get eight heads. That does not change the 50/50 likelihood to 80/20. Observations can be trusted after hundreds of them, not just a few.

The take-a-way of knowing about Hawthorne, bias, and stochastic observations is about knowing how to apply a healthy skeptism of what you are observing. In other words, stick to the null hypothesis until proven otherwise. If you make a change and you observe some favorable results, the finding you walk away with is, 'this seemed to cause that,' not 'this caused that.'

Another thing to remember about the Hawthorne Effect is it was found in an operations setting where work was ongoing. They found an increase in performance caused by a benign variable, that decreased over time. With projects and sprints, you have a start - stop function. Therefore, you will not observe a steady decrease back to baseline. In this type of scenario, what you will likely observe is performance variability of up and down trends that may or may not be due to anything you implemented. You will simply need many observations over quite a period of time to rule out random effects, Hawthorne, or bias.

Answer 2

Based on my experience, you cannot do anything with the improvement ideas of a team, besides giving the team the possibility to implement their ideas, until they support one thing: faster deliveries.

Faster delivery means shorter lead time - you can read this from the CFD - and if the improvements result in better quality, or process improvements etc., then it will be shorter, and it is a good thing to keep an eye on it, so keep doing it. However, finding relation between an improvement and progress is very tricky, because there are too many variables and you may have a false positive result. Here are two examples:

• We introduced code reviews in order to reduce the number of programming mistakes
• We introduced Kanban in software maintenance maintenance

In both cases we observed shorter lead times, but while in the first case it was really the effect of the improvement, in the second case we don't really know what has happened. One says that it was Kanban, others say that it was the new team setup. We tried to implement the same idea with a different team, which failed, and we tried to do something different with the very same team and it also failed.

My point is that when I tried to find a relation between an improvement idea and progress I failed. So, I came up with a list of advises for myself, which has worked so far:

1. I'm not judging the ideas of a team
2. The team is responsible for implementing the idea, the only thing I give is the possibility to do it
3. The team is responsible for measuring its progress. If they measure by themselves it is their data, if I measure it is my data, which won't be accepted by them and nothing will happen
4. I occasionally ask about the ideas, but I never call them to account for doing them or not doing them
5. If the lead time is longer - the idea possibly introduced some waste in the system -, I gently warn them and ask them to do something about it
6. I never use bonus systems to have more ideas or make them happen

My last story. My old manager was obsessed with improvements, and he pushed the teams to have them and implement them. In order to make this manager happy, the teams came up with plenty of ideas, so he was happy. Unfortunately, nobody really cared about the lead time which became twice as longer as before thanks to these fake ideas.

Answer 3

I would use the amount of time it took to establish a sense of velocity as a baseline, then see if the improvement lasts longer than that.

For example, it often takes 3 sprints to get a sense of a team's velocity. See if the improvement lasts longer than 3 sprints. That would seem to suggest that it has made a lasting impact.

Answer 4

I think it's generally possible to see which effects are from which causes in retrospect. Something that I find beneficial is to try and shorten slow feedback loops. Cycle time - the time between getting an idea and getting feedback from trying out the idea - is an example of a feedback loop that's normally pretty slow.

The effects from feedback are just information. Rather than trying to make the effects last - for instance, "That feature worked!" - I would look at what the causes and contexts were that allowed for the positive effect. How many times did things fail on the way? Did we succeed because we were comfortable with failure? What expertise or mindset did we hold that we would like to hang on to? (As well as continously improving, anchoring the things we already hold dear is important!)

Lean and Kanban tend to focus very much on getting things into production, but when you're doing product or software development rather than production line or continuous integration, I think we should be looking more towards learning and feedback as the constraint, and perhaps the goal.

Answer 5

TL;DR

The Hawthorne Effect, if present, is desirable in an agile framework.

Reasonable Question; False Premise

I think the original question stems from a false premise. The premise seems to be that changes to observed behavior as a result of being observed are intrinsically less valuable than unprompted or spontaneous changes.

Continuous Improvement Through Introspection

Agile practices are built around tight feedback loops and continuous inspection of the team's process. The whole premise of agile is that improvement is driven by observing the outcome of a process and making continual adjustments to fine-tune the results.

The Scrum Alliance defines the framework as follows:

Scrum is a simple "inspect and adapt" framework that has three roles, three ceremonies, and three artifacts designed to deliver working software in Sprints, usually 30-day iterations.

When you drive, you don't put your hands at ten-and-two, hold the wheel with an immovable death grip, and hope for the best. You make constant course corrections--generally minor, unless you're careening out of control or making a U-turn, but constant all the same. One might even say that one metric of good driving is the ability to make continual adjustments so that large corrections are not necessary--such as to avoid the fast-moving sequoia that just jumped in front of your car.

Scrum bakes a lot of this into the various required meetings, but the Sprint Retrospective is the epitome of inspect-and-adapt. Since process is made up of people, process change is often dependent on behavioral change to succeed.

Continuous Re-Inspection

Because Scrum is about continuously inspecting the process, and continually adapting the process to current circumstances, there are two obvious conclusions one can draw.

1. Behavior will be constantly observed. Unobserved behavior is an indicator of a hidden process, and needs to be made visible.

2. Behavior change does not need to be permanent, or retroactively measured to any great extent. What matters is whether the current process is supporting the team's current objectives.

Whether the reason is the Hawthorne Effect or some other social driver, the constant adaptation to observation is a clearly-stated goal of agile frameworks such as Scrum.