Keep in mind that no framework or process can guarantee success. However, an agile framework will allow you to adapt more rapidly, and to "fail fast" if the project is not going to succeed as initially conceived. It will also provide more flexibility in determining how to implement the desired system behavior on a new stack, which is essentially what you're doing, and identify areas faster where the original plans for the product and the project need to change.
Regardless of what you currently think, this is essentially a net-new product that will doubtless seek to preserve some aspects of the legacy product it's replacing. Treat it as net-new, and you're likely to be more successful than if you treat it as a modest update or incremental improvement of some kind.
Analysis and Recommendations
What methodology to choose for rewriting an existing software, incremental or waterfall?
You haven't explained why you're re-writing the software. Oh, you did mention that you're trying to port a monolithic (and presumably crufty) legacy app to a new framework or platform, but you don't explain why that adds business value, or what you hope to gain from doing this.
Here are some rules of thumb:
- Refactoring legacy software is generally easier than rewriting it.
- Porting legacy software "whole hog" is generally easy than redesigning it for a new platform or stack.
- Redesigning means re-architecting, re-engineering, and re-baselining all of your assumptions, even if you had 100% of your requirements and 100% of your legacy application under test. Hint: we both know neither of those things are true.
That means you're going to be faced with a lot of unknowns, both known-unknowns and unknown-unknowns, neither of which are conducive to waterfall-style project delivery. There are no guarantees with an agile framework either, but you will certainly discover what you don't know faster with agile practices than you will with waterfall, and spend more time building things than filling out risk logs.
That said, you should consider the project a net-new product with the (possible) goal of emulating some of the behavior of your current system. By focusing on what the current system does that you want to preserve, and what you don't, you free up developer time to focus on how to implement those essential features (along with any new ones) on your new stack.
Just so the point isn't missed, I'll say it again: focus on what you want the net-new system to do, and let the developers work in relatively short feedback loops (1-4 week Sprints, depending on a variety of factors) on how they will deliver those things. The team should focus on delivering cohesive increments of the product that you can collectively inspect-and-adapt each cycle, so that the empirical process can be harnessed to determine where porting, redesign, or simply taking an alternative approach can be leveraged.
You will not only need to continually adjust your project plan, but also continually refine your expectations of the product. Some things work equally well as desktop applications or cloud-based micro-services; others don't. You are about to embark on a journey of exploration, so stop thinking that you or the developers already know what needs to be done. If anyone involved is sure they know everything about the project, they are in for some very unpleasant surprises along the way. Plan on change; in fact, embrace change and treat this as an empirical process where the best path forward at any given point will result in emergent design through an organic process.
In a word, be agile: keep your feedback loops tight, inspect-and-adapt constantly, and strive for continuous improvement. Many IT projects fail; the trick is to minimize expensive or time-consuming failures, which is what Scrum and other agile practices are optimized for.