Is it possible to be agile with a highly technical project?

Question

I'm working on a small (4-6) team on a project that involves creating a machine learning model to predict the outcome of sports matches real-time. This also involves scraping the data and storing it in a database so we can make real-time predictions. We also will be building a simple website to display this data. The client currently has their own popular sports blog, so they plan for the site we're creating to either be an addition (likely just another tab) to their website or a separate site entirely - the client did not specify which, since it likely depends on the quality of our predictions.

Ignoring the user-facing website aspect for the moment (since the website can definitely be done agile-y), the technical aspect alone could take months. Given a deeply technical task like this, is it possible to use agile? It doesn't seem like demoing the technical work done every sprint would be useful to the stakeholder or users, and I don't know what/if requirements could change. Is this task better suited for waterfall, or is there a way to be agile?

Answer 1

Everything in your question suggests that you are exploring a new problem, not simply creating something you already know exactly how to build. Because waterfall asks you to create your design completely before you start building it, using waterfall would be inherently problematic.

In Agile, and Scrum specifically, the goal of making a product increment and putting it in front of users is to learn if you are making the right thing. It is a framework designed for complex adaptive problems and that seems to fit your situation perfectly.

I don't know much about your project, but it sounds like the primary measure of success will be the quality of the predictions. So, for you, I would ask who will judge that and who has interest in that and that person is the most important stakeholder in your review (or people). I would expect your sprint goals and your backlog items to be less

As a [user], I want to be able to click X and see Y

and more

As a [user], I want the algorithm to consider the player's past receiving statistics by quarter so that it accounts for late-game fatigue in the prediction

Each sprint, you would want to look at how you've improved the quality of the predictions and what is the right next work to do in order to make them better. I wouldn't worry too much about the user interface. My personal approach would be to get a basic interface out of the way right away in the first sprint that's backed with a dumb algorithm (it can be a coin-flip for all it matters at first) and then that interface just produces better and better info as you go. That said, an argument could be made that the interface is so unimportant that your user doesn't care about it and you don't even bother with it for the forseeable future. My guess is that if you can make predictions with a high level of accuracy, it can output to a text file for all your user cares.

Answer 2

Validated Learning

Agile approaches involve validated learning. Wikipedia defines the steps of validated learning as:

1. Specify a goal
2. Specify a metric that represents the goal
3. Act to achieve the goal
4. Analyze the metric – did you get closer to the goal?
5. Improve and try again

This actually maps extremely well to an iterative development model such as Scrum, and is incorporated within the standard events of the framework such as Sprint Planning and the Sprint Review.

Why You're Struggling

It is likely that you are struggling because your team is not doing all of the following:

• decomposing tasks sufficiently,
• defining discrete goals for each iteration,
• defining concrete metrics and a Definition of Done to measure each increment, or
• leveraging a well-defined inspect-and-adapt cycle at the end of each iteration.

The Scrum Guide lays out a solid framework for doing these things, and modern practices regarding backlog refinement are covered in books like User Stories Applied, and the use of a Minimum Viable Product (MVP) are covered in books such as The Lean Startup.

Caveats and Next Steps

No single book will give you all the answers you need, and this site is not a book recommendation service. Nevertheless, this should get you pointed in the right direction.

In the meantime, if you have concrete questions about how to split up or decompose a given user story, you are certainly welcome to post that as a separate (and detailed!) question if you need a more pragmatic answer for a specific aspect of your team's challenge.

Answer 3

I found Kanban approach significantly easier to use than Scrum in a R&D kind of a project (were sucessfully using Scrum for 5 years before in other projects). And it works when we also have no single piece of UI. You can demo what you can, the way you can (you can use BDD test report or other reports) whenever you have a produxt increment (milestone). When most of your work are tech spikes (at least in the begining this is going to be the case in an ML project) focus on goals, limiting work in progress and timeboxing the spikes (e.g. review the approach after 2 days of analysis and initial effort not to dive into a rabbit hole)