Does Kanban apply well in practice to software projects with data science background?

Question

A more or less new trend which I observe is integration of data science processes (i.e. workflows required to setup and execute experiments) to design and delivery of user facing apps.

So I assume business expects this new data science capability to introduce some new features either competitive advantages as a new product element.

"Companies approach machine learning, and deep learning, as if they are doing a software engineering project, with waterfall processes and gantt charts, thinking that they can somehow totally avoid the very difficult work necessary to design a complex and highly accurate machine learning solution." (Charles Martin)

Agile seems be of neither help:

"The traditional agile/scrum methodology combined with JIRA may not address the nature of it being a series of hypotheses driven experiments. Time estimates do vary over different projects based on nature of business questions asked. Perhaps, a case-specific discussion along with Kanban style milestone based progress estimation may help." (Kirti Chawla)

P.S. yet a quite clumsy formulation from a recent LinkedIN discussion, please if possible leave a comment to help get this straight (if possible) before just downvoting. Thanks.

So does Kanban help here? Or what is other proven process model? There is also a DevOps approach in this context, but this might be considered as "too technical/low-level" from PM perspective I assume.

Answer 1

I can try to provide some information here. It is almost impossible to say if Scrum, Kanban, or DevOps is applicable to your exact situation, but I can see benefits of any of these.

Kanban

The goal of Kanban is to simply improve process. There is no process that is not potentially benefited by Kanban. The key steps here are to visualize your workflows that you mention in your question and then using a taskboard or some other mechanism, watch as work items flow through those steps. Note that you want the value-delivery item (analysis of a business question, for example) to flow through the steps, not individual tasks. This, plus the other rules Kanban puts into place, like limiting work in progress, will show you where your bottlenecks and inefficiencies are in your process. Then, try small experimental improvements to change your process for the better.

Scrum

Scrum is great at iterating on a problem. Let's say you're trying to improve market impact with your data analysis. Scrum is going to ask you to plan a small targeted analysis inside of a sprint that changes and advances your view of the problem. Then you'll decide what the right next analysis is to take it further in the next sprint. There is a lot of potential here, but some of the common practices that have arisen in scrum are focused on product creation and you might need to leave those pieces behind.

DevOps

The problem that DevOps solves is that software teams were calling things "Done" but those things had to go through a long process before they were deployed and "Accessible" by the person using it. If you have this problem, that work is "done" but not accessible by the people who use it for some time, this might help you. Now, the tools that most people are using probably won't if you aren't building and deploying software, but the ideas and techniques around the processes in DevOps are probably applicable.

Answer 2

Kanban is only vaguely related here but in short, yes it could be useful for all the following model.

It is true that many data science projects are poorly engineered in a waterfall style. They have a technical focus and take a long time to churn out defective software that doesn't meet the users needs and requires considerable and repetitive rework. Kanban can help here with shortening the feedback cycle and reducing the timeline. It can also help move to the next level by reducing the batch size.

I find even waterfall-style data science projects result in a DevOps style of collaboration, certainly from the "test" phase of the project as the end users require an operational environment within which to test the application while the developers are constantly generating fixes that have to be constantly deployed.

The next level is basic agile in the technical part of the process. Here we move on from doing entire technical layers one after the other, to identifying an end-to-end work package. Typically "report oriented". This is related to the fact that most of the data science projects are due to legislative compliance and it is usually specified exactly which reports the organisation is expected to produce and with what content. Here it is possible to identify at least parts of reports that can be fully implemented and tested in less than 2 weeks.

The approach described above, using hypothesis driven experiments as the work-packages is highly advanced and seldom seen even in mainstream agile software development. It does form a part of lean startup though. It requires bridging the gap between "business" and "IT", and creating truly cross functional, market facing, product oriented teams that have all technical expertise to design, deliver and operate their business solutions. In such a data science project we would have one team perform all tasks from the whole value stream. From formulating the hypothesis and experiment, through implementing it in the technical systems, followed by interpreting the results, presenting them to stakeholders and advising in next steps. It truly is the holy grail of data science and extremely effective if you can bridge the organisational hurdles.