Managing huge experimental decision by a developer

Question

The project which I'm working on was over, but I would like to reflect on managing a huge experimental decision made by one developer during development. This is what happened:

A developer strongly felt that something is not right with the current design of the project. Due to insufficient knowledge, he was unable to express out what is wrong. As we have deadlines, we cannot wait, nor work with him on his research as he cannot convince us what's wrong.

To be constructive, he told us to move on while he conducted his research and experimented huge redesigning works on his branch of the SVN. A couple of weeks later, he presented to us a working, hugely redesigned (and refactored) project which he's confident of and explained to us everything. We were convinced.

However, as we were changing and adding features based on the old design, it was as good as we were working on 2 separate projects and SVN merging became impossible due to the large difference. We had 2 options then.

1. To work on our copy, which we would have to redesign all our features to the new standard as well as identifying all refactored stuffs.

2. To work on his copy, which we would have to first identify changed features and implement back correctly in his design, followed by the new features.

We chose the first option, as the second option would still have to deal with lots of redesigning due to the changed features using obsolete classes and the likes.

Lots of time was spent, and everything felt like double (or perhaps triple) effort. Though it was worth it (we did meet the deadline through more effort), question is, given the circumstances, is there any methodology we overlooked that could have more properly handled such scenario? Or was what we were doing the correct decision and that this is actually a normal / common thing in development?

Note that he wasn't that confident of what he was doing either, so he had to finish everything that he was doing to see it working and confirmed his research before presenting it to us. Admittedly we were all amateur developers, including him, but he kinda instinctively knew what's wrong, but lack the terminologies and experience / knowledge of case studies to convey properly to us prior to his research. There was a chance his instincts could be totally wrong too.

Answer 1

TL;DR

Your question touches on a lot of topics, included areas of architecture and engineering which are not project management questions. So, I'll mostly limit my answer to the project management perspective.

The short answer is that you almost always learn lessons during a project that make earlier, initial choices suspect. Hindsight is 20/20, and this is an inevitable consequence of the Cone of Uncertainty. This is why iterative project management and development methodologies that leverage emergent design are generally preferable to waterfall planning, as the cost of a full redesign is often untenable for a typical project.

In cases where a redesign is required, there's no canonical answer to how to do that; it's a matter for each team and each project to consider based on specific circumstance. Beyond that, focus on working software that is "good enough" rather than idealized, and ensure that you are folding lessons learned back into your team's processes.

"Good Enough" Software

There is nothing wrong with time-boxed story spikes, and rework and refactoring are de rigueur for iterative development. However, the YAGNI principle and "good enough" engineering argue against big-bang redevelopment unless your product is unfit for purpose or has accreted too much technical debt to be modified or maintained.

Perfect is the enemy of the good. If your software is working and fit for purpose, it isn't good project management to discard it in favor of something better just because it is better. There should be a valid business driver before project resources are spent on such an endeavor.

Improve Development Processes

While marginally out of scope, I would suggest that your development practices are suboptimal. You should work with the technical team to drive better practices, such as:

• Moving to a source control system that handles branching and merging better than SVN.
• Use techniques such as feature branches and rebasing to enable teams to explore ideas or make changes that can be merged without requiring massive reintegration.
• Make granular, progressive commits with good commit messages so that changes can be cherry-picked between lines of development.
• Use test-driven development to ensure that you can refactor code without changing its external interface or its expected behavior.
• Use automated acceptance testing to validate user-facing requirements and to define expected behavior at the application level.

This is not intended to be an exhaustive list, as good engineering practices are outside the scope of project management. As a project manager, your job is to work with the team to ensure that they have the training, resources, and administrative/technical controls to deliver the product effectively, but how they do it at the technical level isn't your problem...unless they aren't operating effectively. If that's the case, you need to help the team and the stakeholders define a more effective process.