How can we mitigate uncertainty about project estimation in the initial phase?

Question

According to Boehm’s Software Engineering Economics, during the project initiation, estimates can be off by as much as 400%.

However, in order to make the sale, account managers are expected to make on-the-spot estimates of the project duration. The result is that we end up with a number of unrealistic schedules which we are contractually obliged to stick to.

The schedules tends to be underestimated, since there is a feeling here that if we quote too high, the project or the tender will simply be snapped up by another company.

What strategies exist to mitigate that uncertainty? Are there standard contracts that sell an initial requirements extraction phase to better understand the requirements before making a long term estimate?

Answer 1

The way I've been coaching business leaders on is to separate schedule from scope. If you try and completely hit the plan for both you will A- Fail almost for certain, B- Spend a lot of wasted time trying to estimate. This has been true even back to the old traditional project models. The PMP test teaches us about "Order of Magnitude" estimating for a reason.

So I coach the following- Pick the Release Date, "Really" rank order the backlog, Estimate the Backlog, Commit to the top 25% of the backlog, revise at 25%, 50% and 75% of the release schedule.

1- Pick A Release Date: Decide when you need to ship. For most companies predictability of release date is more important than content of release. Knowing they can count on a release going out on X date with high quality (quality is assumed, but does need to be planned), then they are going to be a lot happier and more tolerant of not knowing "exactly" what they will get. Scrum is great for hitting a release date. You know when to ship, you know how many sprints you have and you can plan your scope to that.

2- Really order the backlog: No really, we mean it. You need to stop doing anything with Priority. You can't have five P0s. That's like saying "we have one surgeon and all five of these people have to be operated on first." In trauma medicine we have the concept of triage. If doctors and their staff can make the very hard call on which patient to operate on first, we can make the hard call on if Feature A has more or less business value than Feature B. The key thing here is to estimate on business value AND the level of effort to build. If a story A costs 3 points to build and story B costs 8 points, engineering is probably going to build the 3 point story. If, however, Story A is worth 1 point and Story B is worth 13 points, then Story B probably should get worked first.

Commit to the top 25% This is not a hard number, go with what works. The concept here is that once you have a backlog you can probably have a certain confidence in being able to absolutely deliver a certain amount of the backlog. Don't ever do this as a stretch goal and never more than 50% of the backlog. What you're doing is saying "No matter what, we will ship this, you can take it to the bank." So be cautious.

Revise at 25%, 50% and 75% Then comes the order of magnitude estimating with agile. As you move through the release, you will get a better idea of your velocity and ability to deliver. At these milestones you update what you can commit to deliver based on the reality of what has already been delivered.

So in a year long program, at the start of the program sales knows 25% of the features and can be selling those right then. Three months before ship, you know for certain everything that will be in the release and groups like customer support and professional services can do final training.

Don't try to commit to schedule and scope at the start of a release. It will be fiction and everyone will know it.

Answer 2

Estimating a project and pricing a project are two separate and distinct processes. In IT, at least based on what I have observed, we conflate the two.

Your stated problem is the latter process, in that there is pressure to quote the project--in both price and time--as low as possible due to fear of losing the opportunity. However, you are trying to solve it by doing something...magical on the estimating side. You can mitigate all you want but you will ALWAYS have variability in results because work is ALWAYS probabilistic. And no matter how well you mitigate, pray, hope, wish, do magic, you cannot cure the fear of losing a bid and, thus, essentially giving your services away during the pricing process.

This does not happen in, say, construction. Construction firms do not give away services like IT firms do. The cure is NOT to try to figure out to somehow fit an estimate into an unrealistic price. The cure is to figure out how we in IT can start to behave like other industries and understand that it is okay to let some other firm win the business and lose their shirts on the deal.

Answer 3

There are usually fixed price and cost related contracts.

If there is a given and accepted uncertainty about a project, the customer might accept (or the vendor neglect) a cost related contract. Using this kind of contracts, the customer pays for the real costs independent of the results. Additions to increase motivation are possible also, see e.g. cost plus contract.

The project risk is on the customer site using cost related contracts. Because of that, a customer will use such a contract (in general) only for special projects, e.g. research projects.

Mixed contract types are possible also.

Answer 4

What strategies exist to mitigate that uncertainty?

Here are some:

• Before the Estimate: Provide more training, support, and better process to your account managers. Reduce the expectation of "on the spot" estimating that you mentioned. Figure out exactly how much time they actually have to produce estimates, and make sure that they take advantage of all the time and resources that are available to them, rather than succumbing to the pressure.
• After the Estimate: Use a "time and materials" contract, where possible. In that case, the risk will be mitigated by shifting the consequences. The consequences of having underestimated on a time and materials contract are embarrassment, rather than financial. (for the provider)
• During the Project: Manage the stakeholders effectively, advising them, for example, when your prediction of the actual effort to deliver changes. Make sure they know, in advance, what the impact on them will be. (e.g. difficulty for the client, embarrassment and/or cost to the provider)

Are there standard contracts that sell an initial requirements extraction phase to better understand the requirements before making a long term estimate?

Yes, and no. It depends on what you mean by "standard":

• Yes, some customers and providers may have such options that they consider standard within their company.
• No, I don't believe you will find an industry standard

(I should add that, when you're talking about a large government client, or other client who is dominant in their industry, their internal standards may effectively be a lot like an industry standard.)

Answer 5

I would attempt to mitigate at the relationship level if at all possible. I've had some success with getting a client to be more directly involved in steering the project at a high level, combined with a shared backlog/transparency and agreeing that scope must be flexible in order to make truly high-value product. This is in a fixed-cost, firm (not fixed) date environment.

In these cases, we regularly review the state of the product with the client and then groom the backlog together; things like client feedback, user testing/feedback, and new ideas are prioritized, the backlog is re-chunked into key dates or checkpoints based on our current best-known velocity, and everyone agrees on the changes. A key component is a very streamlined and easy-to-manipulate backlog, preferably in a real-time or shared view. Passing backlogs back and forth tends to promote an adversarial or negotiation approach rather than true partnership. :)

In almost all cases, after these checkpoints and adjustments, everyone seems to be of the opionion that we're much better off than if we had followed a fixed-date/fixed-scope approach.

Answer 6

The most effective way I've found is to have someone from the delivery organization be responsible for writing much (if not all) of the proposal - and at the very least be in a position to have to "bless" the proposal, with the real option to veto a proposal which is unachievable. I never got the chance to implement my other solution (to have sales bonuses held until the end of each project, tied to engagement profitability and estimate accuracy) - would be interested to hear of organizations that have made such a tactic work.