I am in charge of development, and currently my boss and I disagree about the planning approach.
Either you're in charge or you're not. If you're in charge, that means you are responsible for how you implement your process, and will be held accountable for the outcome. If your "boss" is telling you how to implement the development process, then you aren't in charge. It's just that simple. The two of you need to have a conversation about that before you discuss anything else.
The only measure of successful project management is whether the currently-expected outcome is optimal for the situation. You can't guarantee successful delivery, no matter the framework. However, you can create transparency and visibility throughout the project life cycle to support data-based decision-making about the net present value of continuing the project, with or without changes to the plan or one of its constraints. Focus on communicating about those things, rather than on who is right in a more abstract sense.
Aside from the issue above, you are fundamentally disagreeing about whether to:
- do big, upfront design (BUFD, a.k.a. BDUF) using traditional waterfall planning based on a fixed schedule with fixed scope; or
- perform some sort of incremental or iterative development using milestones (implying fixed scope) with schedule being your flexible constraint.
Aside from the methodology or framework where there can be many considerations regarding which is "best," the core issue is that you are
fundamentally disagreeing over which constraints are going to be flexible. The first item above forces either cost or quality to be the adjustable constraints, while your approach makes time (or "schedule") and potentially scope the flexible vertices. Consider the following triangle, where quality is an implied slider.
The Project Management Triangle (Public Domain)
Each project exists within a context, so which constraints should be flexible is situational. It's ultimately a business decision, although process should certainly be a part of the discussion. As an example, if you are in a business such as a startup where there's a fixed burn rate and you have a contractual obligation to deliver a product by a certain date, then schedule must be fixed although (by definition) at least one other vertex must therefore be adjustable to meet a fixed schedule.
Your preferred approach is more iterative, and aligns with the Manifesto for Agile Software Development which values "[r]esponding to change over following a plan" in order to adjust for inevitable changes in the project due to the cone of uncertainty that make BUFD and waterfall planning pragmatically unsuitable for most software projects.
In that sense, you're both right, and you're both wrong. There's no "one true way" to do this, but you're talking past each other in terms of why one process is better than the other for your situation, and discussing the trade-offs in a collaborative rather than an adversarial fashion.
Statistically, more than 68% of IT projects fail. While the failure rate is anecdotally better for agile projects, the more important and more provable point is that product development using iterative and incremental processes are usually:
- More adaptable to changing requirements.
- Offer opportunities for inspection and adaptation throughout the process, rather than only at the end.
- Provide a predictable delivery cadence of working features, which not only ensures better stakeholder alignment but also helps ensure that the project is building something fit for purpose.
- Include test-first design principles, enabling quality to remain higher than a "test at the end" approach which frequently uncovers more bugs or creates last-minute work that can throw off the schedule regardless of how meticulous the planning is.
- Enables the organization to release earlier than planned if the product is sufficiently valuable without waiting until the end of the original schedule; or to pivot for market fit or abandon the project with lower sunk costs than is usually possible with BUFD waterfall projects.
In short, from the point of view of delivering a high-quality product with a predictable cadence of usable (or at least inspectable) increments is much higher with frameworks that support agility than those that don't. That's why agile frameworks like Scrum, the Kanban Method, Lean Manufacturing, and others are often considered better choices in modern software development, especially when emergent design is an acceptable option.
However, there are domains where BUFD is still necessary. For example, spacecraft or medical equipment frequently require fixed specifications or designs that can't be easily changed once development is underway. Voyager, on its way out of the solar system, can't be changed post facto. Likewise, a medical device like a pacemaker must meet exacting tolerances where design, specifications, or quality beyond the research phase must essentially be fixed.
Which to Use When
Here are some very high-level considerations to select between the approaches.
- Use iterative and incremental development when emergent design, just-in-time and just-enough planning, and adjustments to scope or schedule offer adaptability.
- Use waterfall or BUFD when the design implementation is fully known up front and the risk of change is low (e.g. in machining or assembly lines), or when schedule must be fixed but scope, cost, or quality can be adjusted to meet a predetermined schedule.
These are certainly oversimplifications. Entire books are written on this subject, so an exhaustive answer is not possible. However, this is a reasonable approximation of the two schools of thought you and your boss are currently expressing to one another without using the project or business terms of art that would support either position.
Communication is Key
Since you are both currently talking past one another, and both of you want to be "right," your problem is communication. Neither of you is right in an absolute sense. Rather, you could both be right, or both be wrong with the optimal solution somewhere in the middle. The solution (as is usually the case in project management) is to expand the dialogue to cover:
- The business reasons underlying the choice of framework, methodology, or fixed targets.
- A collaborative discussion about the pros and cons of each possible approach.
- The acceptance of risk by leadership (NB: In this case that's not you!) for mandating a process that isn't suited to the problem domain, or for the risks inherent in any business or project management process no matter how well-managed.
Agile frameworks usually do a better job of managing the risks for #3, but overall the question is more about the trade-offs intrinsic to each process and the RACI matrix of your organization's decision-making. While perhaps not politically palatable, you should never accept accountability for delivering a successful project when you have no control over budget, schedule, or scope. In such cases, the best you can do is track progress and report variances from the original plan up to your leadership, explain the source of the variances, and provide options for them to choose from in terms of how the project's constraints can be adjusted to fit the facts or to terminate a project that can no longer deliver a useful outcome.