Should dev oncall fix P1 issues?

Question

We have a scrum team, and are debating two approaches for fixing P1 production issues.

For P0 issues, whoever is the expert in that area jumps in and fixes the issue, so that the issue can get addressed at the earliest. The P2 issues are moved to the backlog and get prioritized for future sprints. However, P1 issues are such that we cannot wait till the next sprint for them to get fixed. Hence, we are debating two approaches here:

1. Whoever is the dev oncall fixes the issue; though they may not have the complete context of that area of code; they would get the context from the dev owner of that feature and fix the issue.
2. The dev oncall only does the initial analysis to confirm that it is a genuine issue, and dev owner of the feature fixes the issue.

In approach 1, the advantage is that nothing planned for the current sprint is impacted, and everyone in the team gets to work on areas that they may not have previously worked on. However, this approach will spend more of oncall bandwidth, and if there are more issues, then, we may need multiple oncall during some sprints.

In approach 2, the advantage is that the P1 issues will get fixed at the earliest since the person with the complete context of the issue works on it. However, something planned in the sprint will have to be deprioritized.

Do you see any other advantages / disadvantages of these two approaches, and would you advice to choose a specific one?

Answer 1

TL;DR

You present two choices for your team, but it's a false dichotomy. The underlying issues are really:

• the lack of intra-team collaboration during development, and
• insufficient slack in your process to maintain a reliable cadence.

In the answer below, I reframe your current problem in order to analyze your chosen solutions. I then offer a "top ten list" of things your team can consider to craft a more suitable long-term solution for the project.

Framing the Problem

However, P1 issues are such that we cannot wait till the next sprint for them to get fixed.

With this statement, you describe a "P1 production issue" as something that takes precedence over the current Sprint Goal or the tasks defined for the current iteration during Sprint Planning. That means that you must adapt your planning process and capacity planning to accommodate these ongoing disruptions to the Scrum cadence.

You define two possible options, but neither is truly optimal. Let's look at why.

Analyzing Option One

1. Whoever is the dev oncall fixes the issue; though they may not have the complete context of that area of code; they would get the context from the dev owner of that feature and fix the issue.

This looks like pair programming, collective code ownership, and collaboration, but it really isn't. By doing things this way, you're creating several problems and violating some core principles. In particular:

1. You are doubling your on-call resources, increasing the level of disruption.
2. You are incentivizing quick fixes over real knowledge transfer. No one is interested in exploratory testing or learning something new at 2 a.m. when they both still have to show up at work the next day.
3. You are still encouraging the team to operate as individuals with unique areas of responsibilities, rather than true collaboration and swarming.
4. You are hiding the impact of production issues rather than making the costs to the project and the team visible.
5. This approach doesn't surface risk to the Sprint Goal.

Analyzing Option Two

Your second option is slightly better, but still has issues. You describe the option as follows:

2. The dev oncall only does the initial analysis to confirm that it is a genuine issue, and dev owner of the feature fixes the issue.

This approach has some pros and cons, rather than simply being all downside. Some examples include:

Pros

1. It reduces the level of off-hours effort to triage.
2. It pushes the effort of fixing the issue onto the Sprint Backlog, making it visible.
3. It creates transparency by tracking the issue as a task or chore on the Sprint Backlog.
4. It forces the Sprint Team (including the Product Owner) and the organization to acknowledge the project costs associated with production support.

Cons

1. Doing triage (rather than solving the problem right then) may increase your cycle time for applying fixes.
2. Doing triage-only will increase the number of internal hand-offs and touch-points, which is an agile anti-pattern.
3. It still compartmentalizes knowledge within the team, rather than encouraging full collaboration or T-shaped people.
4. Production issues that impact your Sprint Backlog may require scope or content changes to the Sprint plan.
5. Impact to the Sprint plan may jeopardize the Sprint Goal and force an early termination of the iteration.

Better Solutions

Don't get trapped by a false dichotomy. If you must choose between two inferior options, then pick the one that is least disruptive or that optimizes for transparency. However, the entire organization would be better served by taking a step back and performing a more comprehensive process analysis. Some grist for the mill includes:

1. Switching to test-driven or behavior-driven design.
2. Building pair programming and continuous integration into your development and QA processes.
3. Ensuring your development, testing, staging, and production promotion processes are both automated and effective.
4. Having a solid roll-back plan that leverages source control and feature toggles to avoid the need to troubleshoot production in the wee hours of the morning.
5. Allocating team capacity to cross-functional knowledge sharing directly on the Product Backlog, making it a visible (but necessary!) cost to the project.
6. Consider splitting production support into a separate team or activity from development. Scrum is optimized for development, not maintenance.
7. Build in additional slack for production support during Sprint Planning, so that you aren't routinely pulling capacity from development.
8. Update your Definition of Done to include regression testing.
9. Treat the routine release of bugs to production as a huge red flag that your process may be broken, and needs careful analysis and redesign. Serious bugs in production should be very, very rare!
10. Use transparency and visible costs to educate stakeholders and senior management on the false economies inherent in an unsustainable cadence, technical debt, or lack of budget allocated to sustainable agile practices.

In short, slowing down the pace by introducing pair programming, TDD/BDD, code reviews, continuous refactoring, continuous integration (CI), and other agile practices is essential to increasing quality and setting a sustainable cadence. This comes at a cost, though, in terms of time and manpower. It's up to senior management whether they want to pay that cost upfront, or pay it over and over down the line as an ever-increasing penalty for treating agility as a "go faster" button instead of a quality and fitness-for-purpose control framework.

Answer 2

Welcome to PMSE!

From the Scrum Guide:

• Individual Development Team members may have specialized skills and areas of focus, but accountability belongs to the Development Team as a whole.

So, your goal should be to progressively move away from the notion of "dev owners of features". Also, it is good to set aside bandwidth for handling unforeseen production issues. This will minimize disruption to the sprint. As you fear, "if there are more issues" I would look for ways to tighten up the QA process. So, my vote is for your Approach 1.