It really sounds like the Scrum Master and the Product Owner have both bought into the velocity and utilization trap. Break the cycle.
Dissecting the Product Owner Role
The PO wants to know how many hours programmer X has available during each sprint and exactly what he will do with them.
Not his business in a Scrum shop. He is part of the Scrum team, but he's not in charge of story assignment or team organization. A Scrum team is self-organizing (when it's working well), so this is a no-no.
He wants demonstrations of features completed at the end of each sprint[.]
This is legitimate. A Sprint Review is a standard Scrum meeting, and definitely part of the framework. It should be focused on completed, user-visible features, but that's somewhat of an implementation detail. The critical thing is that the Sprint Review is the opportunity to show the Product Owner and the stakeholders what's been completed during the most recent Sprint.
[I]f an implementation is more complicated than we expected and something isn't done, he considers that we have broken our commitment.
Maybe, and maybe not. There are several issues wrapped up here. Let's look at them.
- The Team should have a "definition of done." If the Sprint Goal has not been met, and the work doesn't meet the definition of done, then it's simply not done. It's never partially-done; in Scrum, "done" for a user story is all-or-nothing.
- The caveat to the previous axiom is if the Team and the Product Owner cooperatively redefine "done" during the course of a Sprint. If you are part-way through a Sprint and realize you will not meet the Sprint Goal or complete a story, then you should immediately involve the Product Owner to discuss your options.
- If the Team is properly executing Sprint Planning, then the Team is committing to the Sprint Goal and a set of user stories. That's why it's important to estimate properly, and not to overcommit.
How to Handle Failing or Failed Sprints
The PO is wrong in making this a blame issue, though. The Scrum Master's job is to use the Scrum process to manage everyone's expectations during and after a failed Sprint.
If a Sprint is in danger of failing, the PO should be involved ASAP. The Product Owner has several options:
- Work with the team to redefine the Sprint Goal or the accepted user stories in order to salvage value from the Sprint.
- Terminate the Sprint early so that a Sprint Retrospective and a new Sprint Planning session can reboot the process now that everyone has more insight.
Neither option should be a "blame game." Having to renegotiate or restart a Sprint is a common occurrence in Scrum, especially with teams that are new to estimating. It is usually a result of some or all of the following:
- Process impediments that were not foreseen or included in story estimates.
- Mistaken estimates of one or more user stories.
- An overly-ambitious Sprint Goal.
- Over-committing during Sprint Planning due to incorrect story estimates, over-estimating team velocity, the "100% utilization" anti-pattern, having stories assigned to the team (rather than the team making its own commitments), or any other reason that places more stories on the Sprint Backlog than can reasonably be completed in a single Sprint.
The 100% utilization anti-pattern is what concerns me most. You say that the Product Owner says things like:
OK, we have 40 person hours of work for Arthur and 32 person hours of work for Candace, and since they're each supposed to be half-time on this project, they should finish that work by the end of the next sprint.
This is a double fallacy. First, the Product Owner is not permitted to estimate work-effort for the team. His role is to prioritize stories, and to work with the rest of the Scrum Team to adjust scope or delivery dates to meet business requirements based on the Team's estimates of the amount of work involved.
Secondly, a successful Scrum Team is self-organizing. If the Product Owner is assigning work, attempting to implement 100% per-person utilization, or otherwise micro-managing the Team's intra-Sprint processes, then the Scrum Master must:
- Educate the Product Owner on the benefits (and limitations) of the process.
- Ensure the Team is not over-committing.
- Coach the Team to organize as efficiently as possible around accepted Sprint Goals. This has nothing to do with utilization, and everything to do with shared ownership and voluntary commitments.
The goal of Scrum is not to maximize feature velocity or resource utilization; the goal of Scrum is to manage the Cone of Uncertainty involved in large or complex projects, and to create a sustainable pace of work based on increasingly-reliable story estimates. In other words, reliability and consistency trump speed; the speed evolves as processes, estimation practices, automation, and work-flow improve over time.