What framework or methodology would you recommend for a Data Science team?

Question

I am a Scrum Master for a team of mostly Data Scientists and some Software Engineers and a Product Owner.

Our organisation has decided that all teams work in 2 week sprints using Scrum. I personally don't believe it's working. The Scrum framework isn't really working for us for the following reasons:

• It's very difficult to estimate because the level of unknowns is huge. For example, you would have a backlog item to run an experiment, the output could and usually is so varied. The output of the experiment literally drives your work for the coming days.
• Even if you can manage to get the team to break down into stories, there is a vast number of dependencies between stories. It's more like a flow chart or decision tree, then a story map.
• The PO and SM are not data scientists, its impossible to really help with the content of the stories.
• The sprint commitment almost never gets finished due to the reasons above.
• The team doesn't communicate in the same way as engineers. They need a lot of time to discuss and hypothesise(i.e. its not what Scrum was intended for), at least this is my experience.
• Planning meetings become unwieldy because of the unknown nature of the work.

What framework or methodology would you recommend for a Data Science team?

Answer 1

It is worth differentiating between what Scrum is (as defined by the Scrum Guide) and what is popularly associated with Scrum.

For example, story points, stories, estimating, velocity, etc. are not a part of the Scrum Guide.

The team doesn't communicate in the same way as engineers. They need a lot of time to discuss and hypothesise(i.e. its not what Scrum was intended for), at least this is my experience.

Again, there is nothing in the Scrum Guide that says you don't spend lots of time discussing and hypothesising. You are talking about Scrum conventions rather than what the framework actually says.

I have worked with data science teams that used the Scrum framework and they spent a huge amount of time discussing their work.

Planning meetings become unwieldy because of the unknown nature of the work.

Then I would suggest you spend more time synchronising and less time planning. The value of a framework like Scrum is to help a team work together in a more collaborative way, supporting each other if necessary.

To answer your question, both Scrum and Kanban can be made to work with data science teams. The choice of framework usually comes down to the personality types of the team members, the nature of the organisation and the type of domain you are working in.

I'm speculating, but I suspect the issue here is that the team is having an approach imposed on them rather than having control over the way they work. The retrospectives and inspect-and-adapt cycle in Scrum is intended to allow the team to tune the way they work until they find a suitable approach.

Answer 2

Kanban, i.e. managed flow rather than timeboxed sprints, makes a lot of sense for research and discovery work for exactly the reasons you mentioned. Perhaps you could still produce the metrics and status reporting to keep your team accountable to management but the idea is to focus on prioritisation and sustainable delivery rather than estimation and iteration. It should be possible to make a good case for this if you can evidence the kind of issues you mention.

Also work on making sure you have an efficient continuous integration pipeline in place. Provided you can release as often as needed then everyone wins if they don't have to set all their expectations on a one or two week increment.

Answer 3

If there is a lot of 'hypothesising' and meeting without fixed time frames - it's more about that the scope of works is not clarified clearly yet. When the scope is cemented and timeline is prepared, but new requirements are coming or changing or meetings are delayed - we only proceed to work on already agreed scope, highlighting the delays for the Customer with options for trade-off, if he wants to deprioritize one feature in favour to another.

You can try SAFe framework (https://www.scaledagileframework.com/# and https://www.guru99.com/scaled-agile-framework.html) - I can highlight it on top of what other responders mentioned before. From many speakers on professional conferences arranged in Russia, this framework was highlighted many times as used in 'laboratory'(this term is used to name a mid or big-size team, focused on product development in not-well-known-yet area as we). Epic -> Feature -> User Story flow. As you've mentioned that 'your PO is not an expert in the subject area' and that there is a 'a vast number of dependencies between stories. It's more like a flow chart or decision tree, then a story map', i found SAFe potentially suitable for use as it has Features and Feature owners, which become personally responsible for the E2E value, which the guys they rely on are experts in subject area and can provide business value + code.

The decomposition in SAFe is: Epic -> Architectural sub-epic (then breakdown to a Feature(s), which are split to Stories, which are split to Tasks in their turn) plus Business sub-epic (then breakdown to a Feature(s), which are split to Stories, which are split to Tasks in their turn).

My personal experience (business analyst on a large enterprise project, where for one of its phases we've used SAFe-like Feature-based approach in order to focus on the delivery of business value for a few, but scenarios-loaded topics). Feature, is owned by business analyst which product's component is the most impacted (aka 'business owner' - responsible for E2E and for business value specifically) plus dev lead (aka 'technical owner' - orchestrates the development of all the stories, i.e. E2E within the Feature). Meanwhile 'business owner' is dependent on the implementation results owned by 'technical owner', 'business' is anyway the main one, as he finally demonstrates it to the Customer (externally or internal overall Product Owner -as in your case) and gathers the feedback. Business Stories under the Feature, each one is owned by the business analyst (responsible for a particular scenario, i.e. functionality according to feature decomposition). Once business stories are described and there is a visible E2E or solid part of it - kick-off for Technical Story Owner/Team is arranged. Technical Stories under the Feature, each one is owned by dev lead (responsible for particular scenario, i.e. functionality according to feature decomposition) - in order to simplify reporting, DEV Team mirrored Technical Feature->Technical Stories, keeping the links to the ID of Business Story (speaking about JIRA). Once technical stories are implemented and there is a visible E2E or solid part of it - DEMO for Business Story Owner/team is arranged. I didn't highlight Epics as in this approach the 'Feature' was a more relevant term. In short: PROS: personal involvement and personal responsibility. CONS: overhead for keeping an eye on technical tasks for a non-technical'business owner'.