How to handle large R&D projects using Scrum (or any other agile framework)

Question

In addition to our regular software development tasks we have some large R&D tasks which we find difficult to fit into our scrum framework.

The main issue is that there is absolutely no user value before the entire project is done, and it is very difficult to break it down.

Say I want to create an algorithm which can tell cats from dogs. We might already have one with a performance of 98%, but we need a new one (from scratch) which can deliver 99%. The main steps will look something like this:

1. Do a literature study
2. Acquire some pictures of cats and dogs. Perhaps also some code to handle the images.
3. Train a algorithm based on the data and test the accuracy.
4. Make some final validation of the results
5. Implement the working prototype into a product.

Step 1,2,4 and 5 are easy and can be broken further down if needed. However, step 3 is way to big and way to uncertain to just be a story by itself. Experience shows that the progress will be something like:

1. Train the algorithm using the de facto method found in the literature study
2. Realize that you need better performance, but might be able to get it by incorporating some special information in this specific use case.
3. Apply tweak #1 which turns out to improve the performance by x%
4. Apply tweak #2 .....
5. .....
6. n tweaks later you have reached the goal of 99% corrected predictions.

However, since adaptation/tweaking is the main part of the work and it's often impossible to predict the number of tweaks needed, the effect of a tweak and the type of tweaks in advance. We think planning a project like this is very difficult.

We understand that a big (of not the biggest) goals of agile development is that you can adapt and change direction as needed - and that is great. But how to you plan/estimate a story like this, were we find it very difficult to asses the workload in advance, and equally hard to break the problem down.

Any feedback is welcome

Answer 1

I also work in a similar environment.

I suggest you:

• Set a maximum number of tweaks (or time box it) and document the possible "tweaks" and the time they are taking
• Work on understanding your team's speed and similar to the previous point, try to create groups of tasks and review the time they are taking

This two Points will help you in understanding how much they can take when similar tasks or tweaks are required you'll be able to have a more accurate picture of how long you'll require.

Answer 2

Agile Software development process encourages relative sizing. ie. How big is a piece of work vis a vis another piece of work. In this case, it sounds like:

1. You have nothing to compare against. So relative sizing cannot really be done.
2. The incremental improvement (scope of next tweak) cannot be quantified.

So you need a reference for relative sizing. For example if you have some numbers from a past project where you compare birds and fishes. That could help.

Alternatively, consider using waterfall or Kanban. This kind of work sounds roughly-right to be planned as a waterfall project.