Iterative tasks in Machine Learning

Question

We are a small team working on a Machine Learning software. Users can upload images which are analysed, and then download a report about items the AI software found in the image. There are currently two different types of analysis being developed, each of which has its own independent ML model that needs to be trained.

The whole product is in an early pre-MVP stage, with only a couple of pilot users. The main objective right now is to get the AI to a point where the results it produces are even correct (in other terms: we're still evaluating the technical feasibility).

At the same time, we need to bring the web application through which users are accessing the software to a point at which it is production-ready, so there are some to-dos in that area as well. In order to do this, we have a product backlog from which we create our sprint backlogs - this part is straight-forward.

We do struggle however to capture the work on the ML model in our sprint backlogs.

There are some recurring steps that need to be done over and over again:

1. Upload test images to the app to run analyses
2. Check if the results are correct (or at least acceptable) - as of now, they are not
3. Adjust the parameters in the ML model as an attempt to improve the analysis (this is the tricky part, obviously)
4. Rinse and repeat

People in the development team are a data scientist who works on the ML model, a full stack web developer to work on the application itself, and a tester. There is a Product Owner and a Scrum Master (that's my role right now).

So the question is: How do we capture the work on the ML model in the sprints?

• We could write a ticket for each of the above work steps, which would then always be the same ones over and over again. Looks too bureaucratic to me.
• We could also sit down with the data scientist who does step 3 (working on the model), document his work in detail, and schedule it. Feels like too much baby-sitting.
• Or just do the model adjustment work completely on the side, and just agree that the developer who's tasked with it is not part of the development team at this point

Again, the goal is to get at least one of the two AI models to a point where the results are correct. The secondary goal is to reach production readiness with the web application itself (we have a fairly solid backlog about this already).

Answer 1

TL;DR

You lack coherent goals for the machine learning track. This is ultimately the problem you need to solve. Your underlying issue isn't really the type of work, or the framework you're using; it's that part of your project is currently missing clearly-defined goals and the ability to routinely measure progress (or the lack of it) towards those goals.

The one-sentence solution to your problem is: Treat machine learning as goal-directed, test-driven work. That's certainly an oversimplification, but I provide more details below that should help you and your teams design a workflow to do that more effectively.

Analysis and Recommendations

There are some recurring steps that need to be done over and over again:

1. Upload test images to the app to run analyses
2. Check if the results are correct (or at least acceptable)

While you could treat these as a rinse-and-repeat process, doing so is most likely an anti-pattern. Each iteration, the ML deliverable ought to have some sort of well-defined and achievable goal, even if the goal is just to learn something that helps follow-on planning (e.g. what many agilists call a "story spike"). Identify the ML goals and success criteria for each iteration, and then build your work plans accordingly.

As an example, it's unlikely you're uploading random images and that you have no Definition of Done or measurable metrics for the expected results. So, flip the script and define what "done" means for that iteration first.

Define Objectives and Key Results

For example, using a BDD framework like Cucumber:

---

NB: The following is simply a basic example for project management purposes, and not intended to be a detailed technical example. You might want to use Cucumber Scenario Outlines if you need more granular information about what the AI is getting right or wrong.

---

Feature: Identify Red Balloons in Images
  With a sample set of 1,000 balloons and 4,000
  other non-balloon images, the system will
  correctly identify only the 100 red balloons
  with >= 80% accuracy. 

  Rule: Correctly Identify Balloon Shapes

    Example: Find at least 800 balloons in data set.

      Given "data set 12345"
      When the model ingests "images" from the data set
      Then it should find at least "800 balloons"

      # Make sure to add some failing examples, and
      # test for both false positives and false
      # negatives. A Cucumber Scenario Outline might
      # be useful for defining multiple success/failure
      # criteria including tests for boundary conditions.

  Rule: Correctly Identify Only Red Balloons

    Example: Find at least 80 red balloons in data set.
      Given "data set 12345"
      When "800" to "1_000" balloons are identified
      Then the model should find at least "80 red balloons"

You don't have to use Cucumber. You don't even have to use a formal test framework of any kind. However, setting a goal for the iteration and then determining how you will measure whether or not you've met that goal, is absolutely essential.

Time-Boxed Development vs. Continuous Operations

When you have "rinse and repeat" tasks, you're usually dealing with operations rather than product development. There are always exceptions, but treating your ML work as evergreen tasks is:

1. Unsuited to time-boxed iterative development using frameworks like Scrum.
2. While potentially suitable to flow-based or operations-oriented frameworks—no kibitzing about Lean and Kanban also being suitable for creative development, please; yes, they can be, but that's not their ideal use case IMHO—having separate frameworks for a pair of teams that are inherently working towards the same product goals is another anti-pattern.

While you can make arguments for the teams being separate, you can also make arguments for the teams to work together so that they're delivering vertical slices of value rather than running on decoupled tracks.

Even if you decide that you'd prefer to have your web development team use Scrum and your ML team to work a separate track using Kanban or the like, you still need to ensure that the work for the ML team meets INVEST criteria. Broadly speaking, that means the machine learning work should be goal-oriented and measurable too. Even pure research projects, which I've previously addressed, can be decomposed and redefined with testing and measurable goals firmly in mind.

Invite the Team to Collaborate on Cadence and Decomposition

Ultimately, the ML team is part of your product delivery cycle. As such, it makes sense to have a conversation with those team members to see how you can work with them to set goals, measure progress, and visualize the work.

You may also want to have a collaborative discussion with your web development team. See if there are synergies that can be leveraged with CI/CD, automated testing, or other aspects of that part of the project that could help the ML specialists think like agile team members.

You should also invite the ML folks to your Backlog Refinement meetings, and either roll them into your overall Sprint Planning events or have similar meetings with them where work can be decomposed, planned, and some Definition of Done defined so that their work is both directed and performed on some sort of predictable cadence.

Whatever your framework, cadence is the heartbeat of agility. It's the inspect-and-adapt loop at work, and failure to define a cadence or provide routine opportunities to inspect, adapt, or redefine the goals and deliverables within a project are among the most common reason such projects fail. At the very least, that's part of the reason the machine learning portion of your project is struggling and why you should address the lack of cohesion between the work streams head-on.

Answer 2

I work with ML teams and the work process is very similar to yours.

The team will create a ticket for each retraining and will use sub-tasks for the steps (upload data, run tests, analyse results, etc.).

Typically the same team member does all of the sub-tasks, but that is not always true. Occasionally two or more team members will collaborate, splitting up the sub-tasks between them. When doing this, the sub-tasks become very useful. They also help when a team member goes off sick or on holiday and has to hand over the work.

We try and get each retraining done within a 2-week sprint, but that is not always possible.

Answer 3

You're trying to fit a type of work into a process model where it doesn't belong.

When you are using a Scrum-based life cycle, the expectation is that you will have iterations of a generally fixed length. At the start of each iteration, the team will be able to commit to a goal that they are reasonably confident in achieving, and then work throughout that iteration toward that goal, delivering an improvement to the stakeholders at least once during that iteration. Then, they will be able to reflect on the product as well as processes and plan their next steps.

However, not all work fits this model. You may have a goal, but the work and effort needed to achieve that goal may be so vague or ambiguous that the team cannot commit to achieving a goal iteration-over-iteration, especially with a fixed-length iteration. You can still create a goal or hypothesis, do some work to build out what you would need, deploy it somewhere for testing, and then observe and measure how it's working. Since you have two models, you could be working on building and deploying one while collecting observations and measurements from the other to get enough usage data to make informed decisions. Because the work is of a more scientific or exploratory nature, you may have a goal for the next step you want to achieve, but it's too difficult to say if you can achieve it and how long it will take you to get to the next improvement.

Since you have two fundamentally different types of work, it doesn't make sense to manage them within the same framework. I would recommend splitting the work up and managing the work independently.