[...] how do you balance the business' desire for some type of estimation for overall effort, and the desire to remain iteratively agile.
Educate them on why a big upfront estimation can be completely useless.
The problem with business people is that they often lack project management skills and don't understand how software development happens. Consequently, they are unaware of the existence of the Cone of Uncertainty and its implications:
- Estimates (e.g. on duration, costs or quality) are inherently very vague at the beginning of a project
- Estimates and project plans based on estimations need to be redone on a regular basis
- Uncertainties can be built into estimates and should be visible in project plans
- Assumptions that later prove to be mistakes are major factors in uncertainty
When you give business people estimates, because they don't understand what's involved, they take it as your promise to deliver them a product, in a specified amount of time and at a specified cost.
If your project is small (e.g. couple of weeks or months), you have good requirements, you have good specs, you know how you are going to build it, you know the people who will build it, and market conditions are stable, then whatever you come up as an upfront estimate might even happen, and everyone is happy. Hooray!
But most projects are not small. Requirements even if good can't 100% cover everything, team members might change, market conditions might change, because you work for a longer period of time not just a couple of weeks or months you have more risks and you are forced to make more assumptions, etc., so an estimate provided today isn't worth much six months from now.
Unfortunately, business people take the estimate as a given and plan all sorts of other things around the product (launch events, using it as a dependency for other business activities, allocating budgets for other things, etc). When, eventually, you don't deliver on time, a lot of their plans go flying out the window. And it's your fault, of course.
You can obviously provide an estimate up front and give business what they asked for, but you will be keeping them ignorant and they will end up making plans based on something that might not happen.
So, educate them:
- on what an estimate is. Provide a range, not a number;
- on what risks exist, what chances they are to occur, and how that will affect the estimate;
- on what assumptions you made and what will happen if they prove wrong;
- on why estimates are almost always wrong (there has been a lot written about the topic);
- and on why, because of the previous items, an iterative and incremental approach works better. Mention also other advantages (there has been a lot written about this topic too).
If you educate them on why you prefer an Agile approach, and if they understand, they will then know what to expect and will collaborate with you when things go off track, and they can then adapt their plans and goals accordingly. If you just give them the estimate as they asked, they will just blame you when things go off track and will insist on you giving them what you promised, when you promised it, because other plans were made that depend in it.