Accuracy of Estimates

Question

Aclear16, on an answer from another question, brought up the notion of accuracy of estimates. In my view, this notion is nonsensical; however, we talk about this often in PM circles.

In an estimating training class I taught, I wanted to demonstrate the effects of random variability on projects, results that occur secondary to the plethora of variables over which we have little to no control. To do this, I literally, with a time watch, tracked my time from door to door, coming and going, my commute to and from work for over a year. I plotted the results:

(Yes, I hate my commute.)

So, if a commute was a project and I was wanting to target accurately, with this historical data as input to our estimating exercise, where does accuracy live?

If I choose "65" minutes, a slightly more pessimistic target over the MODE, and I arrived at 75, is my estimate "not accurate"? Conversely, if I plan at "80" and arrive in "50", is that also not accurate?

So here's the ultimate question: what does an accurate estimate mean?

(This is question that really does not have an answer and likely violates the rules; however, I hope we can entertain this as I think it is valuable to understanding a very important process in project management. I am hoping some good critical thought and discussion will ensue. Thanks to aclear16 for triggering this!)

Answer 1

So here's the ultimate question: what does an accurate estimate mean?

For me, accurate estimation is like the unicorn: people are looking for it, love to have one at home, but actually it doesn't exist. :-)

Instead of accuracy I tend to use likeliness and focus on certainty. Calculating the likeliness, I'm using the distribution of lead time idea from Kanban (create a histogram of your lead times and pick the longest one). Using it on your example, it would give 87 minutes. That's the estimate. Of course the project can be finished earlier, but that's the date I can tell the customer with the highest certainty at the moment.

There is an important thing: data expires. So, when we are using data for estimation we need another technique which sorts out expired data. For example, the > 70 minutes data was gathered when a huge road reconstruction was performed in your neighbourhood. It is over now, so taking that data out is a good move.

Answer 2

So to begin, here is the quote from the other post that I believe you are referring to:

"... in order to increase the accuracy of our estimations."

Which wasn't intended to imply that one could ever hope to achieve a perfectly accurate estimation (which is almost an oxymoron). This doesn't preclude us from trying to get there. A lot of literature, systems of project management, and training has gone into the idea of increasing the accuracy of estimations. Heck, I even took a class on it in college in the context of mathematical estimation.

So here's the ultimate question: what does an accurate estimate mean?

To return to the matter at hand, I believe an accurate estimate refers to an estimate that falls within an acceptable amount of error from the true value. The amount of acceptable error obviously depends on the context. I was attempting to get at this issue when I said that a project should have very firm short term deadlines, and increased flexibility for deadlines that are further out.

This question does bring up a very good point; at times language will utterly fail us. Many answers are going to prefer this term or that term over accurate estimate, but they all are attempting to capture essentially the same idea. And in all likelihood, none of them will fully explain the idea to everyone that hears them.

Answer 3

An accurate estimate would most likely be a range. I think many people get confused between accurate and precise and numbers (especially below 1,000) inherently provide a false sense of precision. So, for example, given the above data I would probably answer the question with a range utilizing standard deviation to create the range. How far out I went would probably depend on the risk involved with the actual being outside the range. For most things, giving a range out 2 standard deviations gives approximately 95% accuracy, which is usually acceptable. If you don't like the low end of the range you can always also give a couple points on the upper end (I'm about 70% certain it will be done by X, 95% certain it will be done by Y, and 99% certain it will be done by Z).

Answer 4

Firstly, I will answer your question technically, and then I will discuss the project management considerations and implications.

Estimates and accuracy - statistical background

The time that a project will take to deliver is, prior to project closure, a statistical distribution, due to project risks.

You can derive various numbers or estimates from that distribution. For example, you can compute the statistical expectation ('mean' or 'average' time) of the distribution.

The accuracy of an estimate hatt is measured by a metric such as the error t-hatt, where t is the actual time that the project took, a value which is only known at project completion.

You cannot guarantee an accurate estimate prior to project closure. What you can do is minimise the statistical expectation of the error. If you know the distribution, the statistical expectation of the time, E(t) gives you what is called an unbiased estimator, because E(E(t) - t) = E(E(t)) - E(t) = 0 (i.e. the average error is zero). All other estimators will be biased - they will have an expected error that is non-zero - but they might have other desirable properties.

Project Management Implications

Time and cost estimates are useful in project management for several reasons, but it is important that they are not misused. For many of these applications, an unbiased estimator is not the most suitable.

Some key uses of time and cost estimates on projects:

1. Helping the executive and programme or corporate management decide whether or not there is a continuing business justification for a project. For example, a project that costs too much or takes too long might not be worth it.
2. Marshalling resources that need to be planned in advance correctly. For example, specialist team members or equipment might be required after certain tasks are completed, and need to be booked in advance. If they are booked for too early, the booking will need to be cancelled or they will sit around doing nothing. If they are booked for too late, the project will take longer than necessary. This also includes planning activities such as launch parties or media events, which cannot always be changed.
3. Ordering tasks optimally, based on estimates of their cost and duration.

For different applications, different estimators are appropriate; there is not one estimate to rule them all.

When conveying project time and costs to stakeholders, simply conveying an unbiased estimate is not enough; instead, it is far better to ensure that the risks are accurately conveyed (for example, through a Risk Register and a Risk Management Strategy) and approved, so that the relevant stakeholders are aware of the distribution that the project might take in terms of time and cost.

In terms of approvals, ideally a project manager should seek to be given tolerances for project variables such as risk, cost, time, and scope. The project manager should seek to optimise the project, but have the approval to go up to the tolerances in new forecasts without needing to seek further approval. This is far better than working to a single estimate or having a contingency without defining how it can be used.

By explicitly taking into account risk, the risk appetite of the organisation can be taken into account. For example, suppose that the project would fail at its objectives and make a loss if your commute takes more than 70 minutes. There is then a significant chance that the project will fail. For some organisations, the expected benefits might outweigh the risk of failure, but for others, it won't. Simply using an unbiased estimate would not get this.

In terms of marshalling resources, a specific date often has to be nailed down in advance. For example, the room for the launch party might have to be booked. If the date is missed, the room will have to be rebooked, increasing costs. In this case, an estimate is needed. One option is to avoid the risk by being flexible on other variables (e.g. include some could have features, but give the project manager tolerance to cut them if needed to make a deadline). If there is no option but to be firm on the date, the best approach will depend on the organisation's risk appetite and the additional costs if the risk of missing the deadline eventuates. A conservative estimate is one way to reduce the risk (i.e. an unbiased estimator stacked towards the estimate exceeding the actual time), but it might reduce the organisational benefits captured (by allowing competitors more time to get in before the launch party).

The trade-off between risk and benefits is an important business consideration, and there are statistical techniques that can help, such as picking the date that minimises the 'expected loss', or picking a date such that with some high probability (95% or 99%), the actual loss will be less than a certain amount.

Answer 5

An estimate is accurate if the observed time/cost falls within your tolerances around the estimate. As noted elsewhere, these tolerances may be more or less strict and this will impact which part of your distribution curve will be used for the estimate. For example:

• For projects being delivered under a firm-fixed price contract your tolerances will be very strict, so you may want to set your estimate at the 95th percentile or higher (in your table your estimate may be more than 80 minutes).
• For projects that are low risk and of low complexity your tolerances will be much less strict, so you may set your estimate at the 65th percentile and go from there.

This relates to the issue of accuracy ("how close did I get to my estimate") vs precision ("what is the expected error in my estimate"). I think a strong argument can be made that at the very start of a project the precision of an estimate is probably measurable in weeks, which should feed into how strict your tolerances are.

Answer 6

Funny, I use exactly the same example of commute time in my training classes. The point I try to explain is that a (single point) estimate can never be accurate (nor precise, which is a synonym I believe), as it would imply a 100% chance of happening. The probability distribution of a well known task (commute) clarifies the variability. Hence the advise to use ranges (Most Likely - Worst Case) and carefully decide what risk you wish to take, how much buffer or reserve you want to take into account etc. (I can explain this further, but the question is about 'accuracy' I believe).

I don't see the point in calling a range estimate 'accurate'; how it was calculated is more important, and how much risk is implied (Worst case only 20% higher or a 100%)?

I'm a bit worried about the notion of "data expires" discussed by Zsolt: taking out the 'road block' data from the commute time example, would imply no other road blocks are possible on your way to work. In light of risk management, I would certainly leave it in (unless you have the time / and money to test drive and measure every possible alternative route for getting to work).

Answer 7

@Aclear has it right - an "accurate estimate" is one that falls within the acceptable margin of error for that particular instance.

If I say the work will take 5 days and it takes 5.5, then within the overall scope of the work that may be an accurate estimate. A) there was no way to determine beforehand 'exactly' how long it was going to take, so 1/2 day on 5 days is probably pretty good.

On the other hand, if I estimate it will take 20 minutes to drive my kids to school and I leave with 20 minutes until school starts, if it takes 25 minutes and they arrive 5 minutes late then my estimate was not accurate. It was outside the acceptable margin of error.

In reality, an estimate is just that, and estimate. Whether it's an accurate one or not really depends on how accurate I NEED to be.