Designing for the future in an agile environment

Question

I started my working life as a database designer, before moving at a later date into project management. Most of my working life has been spent in "waterfall" environments for a variety of reasons.

One aspect of waterfall design is that data relationships should be understood up-front, allowing the database to be structured correctly.

For example, in an electricity utility environment, you needed to know that a customer could have more than one meter, and each meter could have more than one dial, and that each dial related to exactly one tariff, which would have multiple prices over time. This led to a particular database design that supported the future development of the system without major restructuring of the database, and consequently massive changes to the applications (plural) that used that database.

How do we manage a project in an agile environment to "future proof" it as far as is reasonable in terms of being able to support a complex business, if insufficient design work is done up front.

How should a PM reconcile the wish to deliver something quickly, which may require extensive rework to add functionality in the future, versus doing extensive design up front then being able to develop all of the functionality quickly thereafter, with minimal rework?

Answer 1

TL;DR

Trying to "future proof" your data is done by making your tools, processes, and data structures flexible, not by fixing them for eternity. You do this by embracing test-first database design, ensuring your data is normalized and extensible, and that your tools and processes support change. You do not accomplish this by treating your data or designs as if they are written on stone tablets.

The closest you will likely come to future-proofing your product is to design your data for extensibility, not for immutability. Immutable data may have value in some architectural or engineering domains, but is almost never in itself a business advantage. Adaptability should therefore be a key objective for most projects.

Agility Embraces Change, So Iterative & Incremental Development are Features

[I want] to develop all of the functionality quickly thereafter, with minimal rework[.]

Avoiding "rework" misses the central objective of agile frameworks, which is to embrace changing requirements. The Agile Manifesto values and principles include:

• Responding to change over following a plan[.]
• [Welcoming] changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

Agility is emphatically not about getting everything right the first time, or avoiding the need for refactorings, redesign, or even rework. Instead, agility is meant to address sunk costs and the inability to quickly adapt to change. Sunk costs and inflexibility are often byproducts of big, upfront development approaches resulting from the mistaken notion that product development or support are fully equivalent to a manufacturing assembly line. However, even many modern assembly lines (e.g. the Toyota Production Method) now embrace agility or other just-in-time processes rather than going down the upfront planning rabbit hole.

Databases Aren't Special

The idea that data doesn't change or evolve, or that what data is needed or how it will be used should be fixed, is a straw man. By accepting this a priori you are boxing yourself into a corner. Don't do that.

Data, databases, and the applications that consume or rely on them all change over time, and the need for adaptability is just as real on the database side as it is anywhere else. The idea that you can fully know what data you will need in the future, or how it needs to be structured, is provably wrong.

Databases are just as susceptible as anything else to over-engineering and violations of the YAGNI principle, leading to additional build, carry, and delay costs among others. There are a number of books and approaches to agile database architecture, design, and engineering that embrace change, and many application frameworks (with Ruby on Rails as a singularly outstanding example) that allow you to treat database migrations as an effective and test-driven approach to evolving your data or schema over time.

Design for Flexibility and Extensibility

Whether you're talking about data or the applications that consume them, the accepted modern wisdom is to start with what you know, build that in such a way that it can be extended or modified in the future, and then iterate over your design as required by business needs. This applies just as much to database design as to anything else.

There's no magic bullet. The way you implement agility in any product is by not painting yourself into a corner. As just one database-centric example, denormalized data is a way to trade extensibility for speed. It involves a deliberate trade-off where you must acknowledge that you are creating something that may require more work to refactor in the future (e.g. if the data or its representation needs to change) in exchange for more speed today. If you want to reduce the burden of refactoring or re-engineering your data, don't do that. On the other hand, if you have a non-functional requirement today that can't be solved in a way that is more flexible, then you bite the bullet and do what is necessary rather than worrying about a potential business case that no one is demanding that you solve for right now.

This all involves balance. You can sink unnecessary costs now and accept the risk that you will never actually need that feature; or you say we may never need it and accept the risk that you may in fact need to pay for that feature (including any refactoring, rework, or net-new work required) at some point down the road. By simply measuring and acknowledging the risks, you can then find the optimal business approach that balances risk and reward within the appetite of the project sponsors.

See Also

• Agile Database Techniques: Effective Strategies for the Agile Software Developer. Ambler, Scott. Addison-Wesley Professional; 1st edition (February 20, 2013).
• Test-Driven Database Development: Unlocking Agility. Guernsey, Max. Addison-Wesley Professional; 1st edition (February 20, 2013).
• Other books
  • agile database books on Amazon
  • agile data books on Amazon

Answer 2

Really good question. Really hard to answer. Here is my two cents contribution.

Agile showed up as a response to the practices of the late nineties for building software, practices that assumed you could define everything upfront in such detail and plan them out such that it was then just a matter of following that plan to reach a successful outcome. But reality contradicted this reasoning again and again. Agile just recognizes this reality instead of pretending that it can be ignored if you just invest enough effort in the perfect requirements, designs, plans, or whatever.

In relation to that and in the context of this question, two things come to mind: YAGNI and predictive vs adaptive. Both of these things are ultimately about a prediction of the future: you either need it or you don't, or it's either plan driven or responding to change.

The problem though is that people really suck at predicting the future. You never know when sticking to one decision if that will result in a positive outcome or in a bad one. Only in hindsight you know.

Two examples:

• I worked in a company that was using a web framework for their application and were worried that they might have to change the framework at some point and wanted to make the future change as easy as possible. So they spent a lot of time to wrap that framework in their own built abstraction framework that was hiding the original framework. A lot of time to build something very generic and configurable that would allow them to change frameworks easily in the future. Guess what. They never changed the framework. That scary thing that they worked so hard to protect against never happened. The result? A lot of effort, time and money wasted.
• another company I worked for did the opposite. Developers thought they will not change the framework so they just used it everywhere, even in some places that it shouldn't have been used. Improper abstractions made the thing leak all over the place. Guess, what? They had to change the framework. Again, a lot of effort, time and money lost to rip everything apart and redo it.

In hindsight, these two results would lead to a decision to use proper abstractions with separation of concerns, being careful not to go overboard by wrapping everything in yet another application layer.

My question is therefore about the way to manage a project in an agile environment to "future proof" it as far as is reasonable in terms of being able to support a complex business, if insufficient design work is done up front. How should a PM reconcile the wish to deliver something quickly, which may require extensive rework to add functionality in the future, versus doing extensive design up front then being able to develop all of the functionality quickly thereafter, with minimal rework?

You can't know what to do from the beginning. In the beginning you have assumptions, in the end you have information. So the answer to the question is something like this:

• make plans for the information you have, experiment to test any assumptions you may have;
• think carefully to differentiate between information and assumptions;
• when reality and plans contradict, change the plans, don't ignore reality. Pivot when better information becomes available;
• be aware of the fact that people suck at making predictions about the future which introduces risk no matter what (e.g. too much design upfront to "future proof" things carries a risk that your work will go to waste, insufficient design upfront might need lots of rework later).

Essentially, it's all about trade-offs. Some are comfortable to wing it, to test things out and experiment and leave the future for the future. Others need to make some plans to at least have the confidence that you are not just playing it by the ear. Others chose both approaches and do them together.

If "future proofing" is supported by information then do it, otherwise delay the decision until the last responsible moment when you have better information available.

Answer 3

The agile approach does not necessarily imply shortsightedness.

Depending on the problem domain, you may have a very complete understanding (in your example, the industry exists for quite some time, has best practices, etc.) or a very limited understanding (such as a novel idea for a social network).

Very complete understanding

It would be foolish to throw away already available knowledge, regardless of which approach you follow.

In a waterfall approach, you'd create the big plan which needs to handle all the details first, and then implement that plan. If your knowledge is complete and stable, this can be a valid approach (in reality, knowledge is neither complete nor stable, so you have to put up with somehow handling changes although your process doesn't support it that too well).

In contrast, in an agile approach you plan and implement small increments, preferrably starting with areas that are clear and lead to usable results quickly (reap low-hanging fruit). If you already know what you're going to need, you should ignore the "YAGNI" mantra, it doesn't apply to you. Some increments may consciously implement an aspect incompletely (for example, while you design for multiple meters per customer, your tariffs may initially be very simple-minded). The key here is that you don't do stuff in a way that you know will cause problems later.

Limited understanding

If your initial understanding of the problem domain is very limited, a waterfall approach forces you to gather all knowledge at the start, plan according to what you understood, and execute that plan. You may choose to build a prototype as part of the knowledge acquisition phase, which is kind of doing one iteration of development to gather experience for "the real thing". Still this is only one iteration, and the experience gained may not be sufficient for a valid plan.

When you follow an agile approach, you've got the opportunity to gain knowledge, plan and implement the design incrementally in each iteration. Ideally, intermediate results can be used to validate or enhance your understanding of the problem domain. You'd normally start with areas that are relatively well-understood while working on learning more about the other areas either by experimenting with the intermediate results (that's part of the point of the scrum sprint review - it's not about staging a show to keep stakeholders happy, but letting them touch the product and give feedback) or by setting aside some developer time for some old school learning.

Conclusion

For well-understood problem domains, waterfall may indeed yield good results, maybe even superior in terms of resource utilization since iterations do incur additional costs.

But a fact of life is that few domains are really so thoroughly understood that mid-development changes are unnecessary. As requirements change, the costs of revising a complete plan may be significant. If you don't have opportunities for getting stakeholder feedback, you may create a product that perfectly satisfies the requirements as you gathered them, but doesn't satisfy the client because it's not what they actually wanted.

That's when doing things in an agile way can pay off, both in costs and in client satisfaction.

Answer 4

How should a PM reconcile the wish to deliver something quickly, which may require extensive rework to add functionality in the future, versus doing extensive design up front then being able to develop all of the functionality quickly thereafter, with minimal rework?

This is the question at the heart of agile.

You could rephrase the question as:

Is the value we would achieve from being able to quickly respond to change and feedback worth the cost of possibly not taking the shortest path to a finished product?

The agile answer to this question is typically yes. In agile we value responding to change over following a plan.

The reason people follow the agile approach is that they believe:

• Responding to change and feedback helps you to get to a valuable outcome sooner (but not necessarily get to an outcome sooner)
• It is worth delivering incrementally so that value is delivered continuously rather than all in one go at the end

Answer 5

A very good question! The example you give may be best served by a hybrid model. A waterfall approach to the initial requirements design and db build, then moving into an agile approach for the remainder of the delivery.

Answer 6

Agile does not imply that you start completely unplanned and oblivious to your project goals.

DO NOT start with an empty backlog.
In an agile project, you iteratively work on refining and completing stories from your backlog. How are you going to work with nothing there?

The preferred style of backlog items is the "user story", so in your case one might sound like "As an electrician in the field, I need to report all meters of a customer into the App." Usually, that gives a lot of hints both about the backend and frontend, however you can be way more technical in the description ("we need a one-to-many customer-to-meter relation") and still be fine.

Just collect as many small requirements that you can think of ahead of time in some pre-planning meetings together with the stakeholders, before any actual programming work. The overall collection of stories will inform how to best implement the initial ones. However, if you were mistaken in your pre-planning or your first implementation (which you will be), you can adapt along the road.

That is what we did on a project, we started with a large backlog in place. Of course, many stories changed over the course, but the initial phase was a breeze.

Answer 7

In my experience there is two areas where it is a good idea to prepare for the future. The arena is in how you want to create, deploy and update. As you are aiming to work agile you expect to do a lot of changes both to the code and the database. The first area then is having good code management tools and the methods for managing code, this area is sort of well known. As example git supports a lot of concurrent developers working on code each in their own "sandbox" and then pushen or pulled to the next deployment (or release). How to do continous integration builds is a well known technique and is probably part of your test harness.

The situation is less simple in database change deployment. In order to be able to change the database, say modify an existing database field, you might need to create some "deployment function" that modifies existing data and perhaps flags data that might need some specific treatment. Getting this part of the change process automated is the second area where I would explicitily aim for the future -- databases changes will happen and it is very good idea have tools for updating the data along with the database structure and synchronize this with deployment of new code. I guess (hope, believe) that there are tools available for this already, but I have not worked with any. And, as you probably easily can back out from a failed program update, you should probably think about how to back out from the associated data base change (while keeping any good new data).

Answer 8

In my opinion, and in my experience, "in this business situation, first go for Waterfall."

Most businesses – including "keeping the lights on" – involve many complexities that are purposely concealed from the customer. However, any proposed "solution" that does not carefully consider them is simply "an expensive waste of time" which also makes the entire team look very ignorant.

So – here's my pragmatic suggestion:

• Waterfall the requirements discovery process. Then ...
• Agile the subsequent execution of the computer software which will ultimately address [portions of ...] those requirements. Which requirements have by now been "rigorously discovered." So that nobody will now be wasting anybody's time.

Yes, even though "Messrs. Lewis & Clark" once embarked upon what they termed "the Voyage of Discovery,™" they already well-knew ... as best as they then could ... where they were going.