how to do spike/research story which can run more than 1 sprint

Question

I past I have done a research/spike story which was a new area for me. Building big excel, debug ntml2 jar for authentication issue. our sprint was 2 weeks and it runs for more than 1 sprint, what Scrum says about handling such a story and how to timebox it if it can deprioritize. And story point nowhere converts to days. so if the story does not deliver in a sprint, team velocity will go down and management will get worried.

Answer 1

TL;DR

A research spike is intended to reduce the cone of uncertainty for future work. It is not intended to deliver shippable increments of anything. Treating the output of a spike as anything other than input to story planning/refinement is a Scrum implementation smell.

Time-Box Your Spikes

As an empirical control framework, Scrum is heavily reliant on time boxes to create feedback loops and opportunities to inspect-and-adapt. A story spike is really just a specialized user story that provides just enough knowledge to plan a related backlog item. A spike should not be used to contribute directly to the current Sprint Goal, and you should strenuously avoid having same-Sprint dependencies on a spike.

Spikes should have a maximum size of one Sprint, just like any other story. Each of the exploratory tasks for a spike should have an estimate on the Sprint Backlog, just like other Sprint Backlog items. You then handle spike slippage the same way you handle user story slippage: fail early. Tasks that exceed their Sprint Backlog estimates should be communicated in the daily stand-up, and the whole team (including the Product Owner) should decide what to do about spikes that won't deliver value or that can't be completed within their time-box.

Decompose Your Spikes

Spikes must fit within a single Sprint, and should be fairly concrete in their objectives. Building a large Excel spreadsheet, or debugging a complex library, are appropriate targets for a spike. Instead, you should consider the actual planning goals and develop spikes that answer targeted questions. For example:

1. We need to build a large authentication table with 100 unique states, and we don't know how to estimate that story.

   • Let's try building a smaller sample of around 10 states in Excel, just to get a sense of how much work would be involved as we scale up!
   • We'll guesstimate 5 story points for the spike to build the scaled-down sample, and then use the outcome of the spike to improve our estimates for the full story next Sprint.

2. NTLM authentication is broken, but we don't know how much work it will be to fix it.

   • Let's use the error message to find the method call that's a problem, and then tally up all the places in the library that call that method. That should give us a rough estimate of the size of the problem.
   • If we can identify the problematic method, we'll examine it as a team to get a sense of how easy (or difficult) it will be to fix it, work around it, or develop an alternative.

In each case, the goal of the spike isn't to solve the problem. It's to define a problem, validate an idea, or scaffold a quick-and-dirty proof of concept. The only work involved should be just enough research to estimate the size of a related problem with unknown scope or difficulty.

As currently described, you're treating spikes as "deliverable work of unknown size." Don't do that! You should be decomposing your spikes until they are small, targeted explorations that support planning and level-of-effort estimation of future work.

Research is Work

Spikes are work. All work for a Sprint should be estimated, prioritized, and planned. Therefore, time allocated to spikes should be counted against the team's capacity for the Sprint. However, the current Sprint Goal should not be dependent on the spike, nor should a "failed" spike imperil the Sprint. In fact, when properly done a spike really can't fail because its sole purpose is to reduce the cone of uncertainty.

So, setting aside the question of whether you should be treating velocity as a productivity metric (hint: you shouldn't!), a spike completed within its planned time box is "done" and won't impact your real velocity. An incomplete spike that yields learning is also "done," and won't impact your velocity either. The only way a spike will drag on your velocity is if it's not started, or if it was not executed in a way that yields any sort of validated learning that narrows the cone of uncertainty for a future Sprint.

Note that if the spike is somehow "not-done," then a team's velocity metric should be impacted. Otherwise, you're creating invisible work or hand-waving your Sprint Planning. Neither of these things will improve the efficiency or predictability of the project, which requires that potential drags on capacity be kept visible.

Don't protect your velocity metric in a misguided attempt to sweep problems under the rug. Transparency is a requirement of Scrum, and velocity only works effectively as a detective control and capacity-planning tool when its an honest reflection of the work a team can likely complete within a single Sprint. If your spikes are legitimately dragging on the team's velocity, then that's an issue to address in the Sprint Retrospective for future Sprint Planning and Backlog Refinement activities.

Answer 2

The purpose of the limited timebox is to force the team to break down larger items into smaller ones. 2 weeks is a long time for research. I usually look for research tasks to be measured in hours, not days, let alone weeks. You may need to consider exactly what it is you are researching. Some common ways to approach this include:

In pure research, what are you focused on? Are there key questions you need to answer? Are there capabilities you need to prove out?

In debugging, you may want to eliminate possible causes, or build in some triggers that will provide you more information.

In general, if you have a task like this that you expect to take weeks, ask yourself what you hope to know in a day or two. Or, another way to look at it: What could you discover that would tell you if it was worth spending a few weeks on or not.

Answer 3

There are a few different ways to approach this kind of problem. Since this question is tagged Scrum, I'll focus on Scrum as it's defined in the Scrum Guide, but with other lean and agile approaches.

Scrum already accounts for research - it's called "Product Backlog Refinement". At one time, the Scrum Guide recommended that the team allocate approximately 10% of its capacity for Product Backlog Refinement, but does not make this a firm timebox, meaning that the team can choose to allocate more or less time as necessary. In recent editions of the Guide, this has been removed, but I believe it to still be a good rule-of-thumb. Sprint Planning is a good place to account for Product Backlog Refinement activities that the team is planning on in the upcoming Sprint, since you're already looking at the team's past performance and projected capacity in order to determine what Product Backlog Items can be brought into the Sprint and what the Sprint Goal looks like.

The output of Product Backlog Refinement is not a deliverable for a Sprint. That means that you don't necessarily need to deliver something at the end of the Sprint timebox with respect to your refinement. The work to refine Product Backlog Items will be visible in that the Product Backlog gains new details - Product Backlog Items are modified or created (or removed) and additional details are added. It may make sense to talk about these Product Backlog changes at the Sprint Review - one of the defined elements is that the Product Owner discusses the Product Backlog, which implies a discussion of why the state of the Product Backlog is what it is.

During a Sprint, the Sprint Backlog is purely owned by the Development Team. Items may be added to the Sprint Backlog to specifically address questions that need to be answered during research and investigation (Product Backlog Refinement). There are benefits to including work in the Sprint Backlog to address refinement - it can increase transparency (one of the Scrum pillars) into the work that the team is doing and can help the team focus their work and then inspect (another Scrum pillar) what they were able to do at the end of the Sprint.

The implementation of what this looks like with respect to your Product Backlog, Sprint Backlog, and Sprint Goals will vary depending on what tool(s) you use for your work. But it's possible to implement something that supports these ideas in the various tools that I've worked with.

Although tangential to your core question, I would add that management being worried about changes in the team velocity is somewhat troubling. Velocity should not be reported outside of the team. Although some teams find it useful for Sprint Planning purposes, it's a measurement that is very prone to disturbances. Between changes to the definition of a "story point" over time to changes in knowledge of the team to process changes (especially with respect to Definition of Done) to changes in team capacity, it's not something that can be stable. Management should focus on the team continually delivering value to stakeholders, not delivering story points at the end of a Sprint.