What methodology to choose for rewriting an existing software, incremental or waterfall?

Question

We are rewriting and re-designing a desktop application (something like QGIS) into a new Web Based application with new technology.

We almost know 80% of the requirements and use cases because we have a working application. We estimate a 12 month development with 3 senior developers, What methodology is the best for such development?

• Using Scrum with 1 month sprint duration, Then my question is: If we use incremental development while we know in next sprint we should change and refactor those features, does it worth?
• Or it is better for us to start with a waterfall-like methodology (e.g. spend 3-4 month analyzing and designing) and then develop incrementally.

What methodology experts use in rewriting situations?

Answer 1

The safest answer for any software development work is to start and continue incrementally (doesn't have to be Scrum though).

There are some bigger questions implied by what you have said. It's frequently said that "rewriting" an application is one of the things you should never, ever do. Don't start with an assumption that you should have a new solution 80% like the old one. Start with the things that are going to be most valuable or that have the most opportunity to improve on the existing solution, deliver the first iteration and iterate from there based on user feedback. You might be surprised how different the new software turns out.

One month is too long for a sprint duration of a software project and 3-4 months is definitely too long to spend "analysing and designing". Aim to deliver working functionality every two or three weeks, ideally several times per week so you can get feedback and fix defects early.

Answer 2

You are building an entirely new application. The fact that you have access to an existing application which serves the same purposes using different technology will benefit you only to the extent that it illustrates the problem that is being addressed. "Desktop" applications use an entirely different architectural approach than "web" applications do.

Full disclosure: One of the things that I am often commissioned to do as a consultant is to help to prepare development plans for companies who are considering such a move. Because I am ... ahem ... "old enough to remember." And, one of the things that I always try to present them with is – "what if you left it as a 'desktop' application?' But overhauled the technology?"

"Web-ifying" a desktop application invariably involves "a tremendous amount of JavaScript." Which essentially becomes the functional implementation of the application: "it really is 'still running on your desktop' after all." But the complexity has just gone through the roof – and you've lost a lot of control because you don't own the myriad JS libraries that you have to use.

Soberly consider all of your available technical implementation options before you begin to consider team dynamics methodologies. "Poco a poco ..."

Answer 3

I don't have a ton of experience with reimplementing/rewriting an existing application from scratch, but the experience that I do have tells me that you probably don't know 80% of the requirements and use cases. Maybe you know 80% of what the existing system does, but there are still plenty of unknowns. Maybe the existing application has defects or deviations from documented functionality that have become expected behavior. You probably don't have good data about what functionality in the existing application is actually used. For functionality that is used, cloning existing behavior may not be what the end-users need or want.

I struggle to find cases where plan-driven methodologies work for software. In nearly every case, I'd recommend an iterative and incremental approach to get regular feedback on the state of the product and to maximize the value delivered in each increment. Even in a replacement effort like this, you may find that a 1:1 replacement isn't what is truly needed. Finding this out 8 or 9 months into the effort isn't very effective.

Answer 4

TL;DR

Keep in mind that no framework or process can guarantee success. However, an agile framework will allow you to adapt more rapidly, and to "fail fast" if the project is not going to succeed as initially conceived. It will also provide more flexibility in determining how to implement the desired system behavior on a new stack, which is essentially what you're doing, and identify areas faster where the original plans for the product and the project need to change.

Regardless of what you currently think, this is essentially a net-new product that will doubtless seek to preserve some aspects of the legacy product it's replacing. Treat it as net-new, and you're likely to be more successful than if you treat it as a modest update or incremental improvement of some kind.

Analysis and Recommendations

What methodology to choose for rewriting an existing software, incremental or waterfall?

You haven't explained why you're re-writing the software. Oh, you did mention that you're trying to port a monolithic (and presumably crufty) legacy app to a new framework or platform, but you don't explain why that adds business value, or what you hope to gain from doing this.

Here are some rules of thumb:

1. Refactoring legacy software is generally easier than rewriting it.
2. Porting legacy software "whole hog" is generally easy than redesigning it for a new platform or stack.
3. Redesigning means re-architecting, re-engineering, and re-baselining all of your assumptions, even if you had 100% of your requirements and 100% of your legacy application under test. Hint: we both know neither of those things are true.

That means you're going to be faced with a lot of unknowns, both known-unknowns and unknown-unknowns, neither of which are conducive to waterfall-style project delivery. There are no guarantees with an agile framework either, but you will certainly discover what you don't know faster with agile practices than you will with waterfall, and spend more time building things than filling out risk logs.

That said, you should consider the project a net-new product with the (possible) goal of emulating some of the behavior of your current system. By focusing on what the current system does that you want to preserve, and what you don't, you free up developer time to focus on how to implement those essential features (along with any new ones) on your new stack.

Just so the point isn't missed, I'll say it again: focus on what you want the net-new system to do, and let the developers work in relatively short feedback loops (1-4 week Sprints, depending on a variety of factors) on how they will deliver those things. The team should focus on delivering cohesive increments of the product that you can collectively inspect-and-adapt each cycle, so that the empirical process can be harnessed to determine where porting, redesign, or simply taking an alternative approach can be leveraged.

You will not only need to continually adjust your project plan, but also continually refine your expectations of the product. Some things work equally well as desktop applications or cloud-based micro-services; others don't. You are about to embark on a journey of exploration, so stop thinking that you or the developers already know what needs to be done. If anyone involved is sure they know everything about the project, they are in for some very unpleasant surprises along the way. Plan on change; in fact, embrace change and treat this as an empirical process where the best path forward at any given point will result in emergent design through an organic process.

In a word, be agile: keep your feedback loops tight, inspect-and-adapt constantly, and strive for continuous improvement. Many IT projects fail; the trick is to minimize expensive or time-consuming failures, which is what Scrum and other agile practices are optimized for.

Answer 5

Rewriting and re-designing a desktop application into a new Web Based application with new technology ==> That probably means not only coding, but also setting up your infrastructure, development stack, user roles, etc.

Learn how to work agile. You can partially do that as you go, but you have to have your entire company aligned on the method from the start. Do you have people for the different roles (PO/SM)?

I suggest defining the following stages:

• A minimal product that does not really do anything. And I mean minimal: just something like a splash screen and a button that you can click on.
  Split everything you need to do into user stories/spikes, determine their dependencies, and start implementing those.
  When this is finished, you will (roughly) have set up your infrastructure and learned how to work together.
• Some limited functionality, a 'version 0.2'.
• Then gradually build in more functionality. Start thinking about a shippable product: what should it contain? All the time keep it agile, i.e. aim for a potentially shippable product after each sprint.
• Only after 8-10 sprints and/or something like a 'version 0.5', you may be able to estimate how long it will take to deliver version 1.0. Do not give out estimates any earlier, they will be extremely unreliable. This is why you need the alignment mentioned earlier.

All the time, keep training on working agile. Do your retrospectives - find out where you can improve! Make sure your people keep growing in their roles. You'll notice that the machine will get oiled better and better, and you'll get a feel for what you can deliver.

Do not forget that while doing all this, your current product has to be maintained, with occasional new features, because this is what brings in the money. But keep your time investment low there.

We are also following this process in my company. To give you an idea: the software that we have been building for twenty years will have a shippable version with very limited functionality after more than one year.

Answer 6

These types of projects are great candidates to be managed with agile principles; prepare your stories and run weekly sprints. Working with small timeframes and aiming for small gains are always valuable since it shows where we are going.

I recommend you read my article "Why Estimates Aren’t Accurate For Software Development Tasks?" about this.