Agile in Data Science - how can I do it best

Question

I work in a data science team and we're starting along the path of agile development.

We will be working in 2 week sprint cycles.

Once we have a clear view of what we are doing, it looks good - we can break the development out into small stories and deliver them

However, I am not really clear on the research phase of the project. If we need to research the problem, identify possible data we can use to solve it and then explore that data; how can we possibly estimate and break into stories?

I guess we don't know what we don't know - the problem is a new one, it's not been tackled before & we don't know how to approach it, so how do you go about estimating?

Thanks a lot

Answer 1

One solution might be to use Kanban, without sprints, and focus on the importance of the work, its prioritization and order, and less so on building estimates. Building in Sprints can also work, but the focus would probably still need to shift (see also this question).

Another solution would be to split your work into "exploratory" work and "implementation" work. With the "exploratory" work you figure out "what" to do and then create stories for "implementation", where you address the "how" and "how long" it will take. You can run these tracks alternatively (as in the images from this post), or you can run them in parallel.

Answer 2

We will be working in 2 week sprint cycles

One other thing that may be worth considering is having a longer sprint cycle. For example, if your sprints were 4 weeks long, would that give you more time to do the research work and still deliver value within a sprint?

However, I am not really clear on the research phase of the project. If we need to research the problem, identify possible data we can use to solve it and then explore that data; how can we possibly estimate and break into stories?

The key to delivering value in stories is to understand that it doesn't need to be a huge amount of value. A user story can deliver just a sliver of value.

For example:

As a sales manager I would like to know what database tables contain information on new customer leads, so that I can query them to generate a list of leads

Chances are the sales manager will never actually query the tables, they will instead wait for your team to build a suitable report. However, once the data location has been established they could in theory ask somebody to run some SQL against the tables to generate the data they need. In this way the story does deliver some value, even though it will likely never be acted on.

This is a useful approach to building user stories in data science type work. It still commits the team to delivering value and it will produce results that are understandable by your users (e.g. "The team this sprint delivered the knowledge of the location of all database tables that include patient history records.").