11

Aclear16, on an answer from another question, brought up the notion of accuracy of estimates. In my view, this notion is nonsensical; however, we talk about this often in PM circles.

In an estimating training class I taught, I wanted to demonstrate the effects of random variability on projects, results that occur secondary to the plethora of variables over which we have little to no control. To do this, I literally, with a time watch, tracked my time from door to door, coming and going, my commute to and from work for over a year. I plotted the results:

enter image description here

(Yes, I hate my commute.)

So, if a commute was a project and I was wanting to target accurately, with this historical data as input to our estimating exercise, where does accuracy live?

If I choose "65" minutes, a slightly more pessimistic target over the MODE, and I arrived at 75, is my estimate "not accurate"? Conversely, if I plan at "80" and arrive in "50", is that also not accurate?

So here's the ultimate question: what does an accurate estimate mean?

(This is question that really does not have an answer and likely violates the rules; however, I hope we can entertain this as I think it is valuable to understanding a very important process in project management. I am hoping some good critical thought and discussion will ensue. Thanks to aclear16 for triggering this!)

  • 1
    Can you link to the other question in this one? – Thomas Owens Aug 25 '12 at 21:29
  • 1
    I would if I could. Have no earthly idea how. Anyone else? – David Espina Aug 25 '12 at 21:30
  • 1
    Can you provide a link to it? I'll edit it in. I'm just not sure which answer of Aclear16's you're referring to. – Thomas Owens Aug 25 '12 at 21:30
  • 1
    It is the 40 hour week one.... – David Espina Aug 25 '12 at 21:34
  • 1
    Here is the link: pm.stackexchange.com/questions/6362/… – Sam Aug 25 '12 at 23:16
6

So here's the ultimate question: what does an accurate estimate mean?

For me, accurate estimation is like the unicorn: people are looking for it, love to have one at home, but actually it doesn't exist. :-)

Instead of accuracy I tend to use likeliness and focus on certainty. Calculating the likeliness, I'm using the distribution of lead time idea from Kanban (create a histogram of your lead times and pick the longest one). Using it on your example, it would give 87 minutes. That's the estimate. Of course the project can be finished earlier, but that's the date I can tell the customer with the highest certainty at the moment.

There is an important thing: data expires. So, when we are using data for estimation we need another technique which sorts out expired data. For example, the > 70 minutes data was gathered when a huge road reconstruction was performed in your neighbourhood. It is over now, so taking that data out is a good move.

  • 1
    Is there a danger or penalty in providing, or targeting, the longest lead time, e.g., 87 minutes in this example? – David Espina Aug 25 '12 at 22:59
  • 2
    Continually targeting the longest lead time encourages your team to never push themselves, probably increases your costs, and is generally not a good idea. I believe it is as detrimental as continually choosing the shortest lead time. Extremes are usually not good targets. – Andrew Clear Aug 26 '12 at 4:35
  • @aclear16 for me, continuous improvement comes first - with the culture - and its result is reflected in the estimation. – Zsolt Aug 26 '12 at 6:38
  • @DavidEspina the danger I'm aware of is losing market. If I go with 87 and the competition offers 67 for example, people will go for the 67. Even if the 67 version gets postponed several times they'll stick to it (they won't give up their investment) and won't come for my 87. If I give 66 I put my company in risk, because I have to self fund the additional effort caused by the 87-66. And we got to the risk management topic. – Zsolt Aug 26 '12 at 6:47
  • @Zsolt: Can you clarify what you mean by "continuous improvement comes first?" How, if at all, are you fostering this by choosing the longest lead time? Also, are you referring to continuous improvement of your personnel, or continuous improvement to your project? – Andrew Clear Aug 27 '12 at 17:58
4

So to begin, here is the quote from the other post that I believe you are referring to:

"... in order to increase the accuracy of our estimations."

Which wasn't intended to imply that one could ever hope to achieve a perfectly accurate estimation (which is almost an oxymoron). This doesn't preclude us from trying to get there. A lot of literature, systems of project management, and training has gone into the idea of increasing the accuracy of estimations. Heck, I even took a class on it in college in the context of mathematical estimation.

So here's the ultimate question: what does an accurate estimate mean?

To return to the matter at hand, I believe an accurate estimate refers to an estimate that falls within an acceptable amount of error from the true value. The amount of acceptable error obviously depends on the context. I was attempting to get at this issue when I said that a project should have very firm short term deadlines, and increased flexibility for deadlines that are further out.

This question does bring up a very good point; at times language will utterly fail us. Many answers are going to prefer this term or that term over accurate estimate, but they all are attempting to capture essentially the same idea. And in all likelihood, none of them will fully explain the idea to everyone that hears them.

3

An accurate estimate would most likely be a range. I think many people get confused between accurate and precise and numbers (especially below 1,000) inherently provide a false sense of precision. So, for example, given the above data I would probably answer the question with a range utilizing standard deviation to create the range. How far out I went would probably depend on the risk involved with the actual being outside the range. For most things, giving a range out 2 standard deviations gives approximately 95% accuracy, which is usually acceptable. If you don't like the low end of the range you can always also give a couple points on the upper end (I'm about 70% certain it will be done by X, 95% certain it will be done by Y, and 99% certain it will be done by Z).

3

Firstly, I will answer your question technically, and then I will discuss the project management considerations and implications.

Estimates and accuracy - statistical background

The time that a project will take to deliver is, prior to project closure, a statistical distribution, due to project risks.

You can derive various numbers or estimates from that distribution. For example, you can compute the statistical expectation ('mean' or 'average' time) of the distribution.

The accuracy of an estimate hatt is measured by a metric such as the error t-hatt, where t is the actual time that the project took, a value which is only known at project completion.

You cannot guarantee an accurate estimate prior to project closure. What you can do is minimise the statistical expectation of the error. If you know the distribution, the statistical expectation of the time, E(t) gives you what is called an unbiased estimator, because E(E(t) - t) = E(E(t)) - E(t) = 0 (i.e. the average error is zero). All other estimators will be biased - they will have an expected error that is non-zero - but they might have other desirable properties.

Project Management Implications

Time and cost estimates are useful in project management for several reasons, but it is important that they are not misused. For many of these applications, an unbiased estimator is not the most suitable.

Some key uses of time and cost estimates on projects:

  1. Helping the executive and programme or corporate management decide whether or not there is a continuing business justification for a project. For example, a project that costs too much or takes too long might not be worth it.
  2. Marshalling resources that need to be planned in advance correctly. For example, specialist team members or equipment might be required after certain tasks are completed, and need to be booked in advance. If they are booked for too early, the booking will need to be cancelled or they will sit around doing nothing. If they are booked for too late, the project will take longer than necessary. This also includes planning activities such as launch parties or media events, which cannot always be changed.
  3. Ordering tasks optimally, based on estimates of their cost and duration.

For different applications, different estimators are appropriate; there is not one estimate to rule them all.

When conveying project time and costs to stakeholders, simply conveying an unbiased estimate is not enough; instead, it is far better to ensure that the risks are accurately conveyed (for example, through a Risk Register and a Risk Management Strategy) and approved, so that the relevant stakeholders are aware of the distribution that the project might take in terms of time and cost.

In terms of approvals, ideally a project manager should seek to be given tolerances for project variables such as risk, cost, time, and scope. The project manager should seek to optimise the project, but have the approval to go up to the tolerances in new forecasts without needing to seek further approval. This is far better than working to a single estimate or having a contingency without defining how it can be used.

By explicitly taking into account risk, the risk appetite of the organisation can be taken into account. For example, suppose that the project would fail at its objectives and make a loss if your commute takes more than 70 minutes. There is then a significant chance that the project will fail. For some organisations, the expected benefits might outweigh the risk of failure, but for others, it won't. Simply using an unbiased estimate would not get this.

In terms of marshalling resources, a specific date often has to be nailed down in advance. For example, the room for the launch party might have to be booked. If the date is missed, the room will have to be rebooked, increasing costs. In this case, an estimate is needed. One option is to avoid the risk by being flexible on other variables (e.g. include some could have features, but give the project manager tolerance to cut them if needed to make a deadline). If there is no option but to be firm on the date, the best approach will depend on the organisation's risk appetite and the additional costs if the risk of missing the deadline eventuates. A conservative estimate is one way to reduce the risk (i.e. an unbiased estimator stacked towards the estimate exceeding the actual time), but it might reduce the organisational benefits captured (by allowing competitors more time to get in before the launch party).

The trade-off between risk and benefits is an important business consideration, and there are statistical techniques that can help, such as picking the date that minimises the 'expected loss', or picking a date such that with some high probability (95% or 99%), the actual loss will be less than a certain amount.

1

An estimate is accurate if the observed time/cost falls within your tolerances around the estimate. As noted elsewhere, these tolerances may be more or less strict and this will impact which part of your distribution curve will be used for the estimate. For example:

  • For projects being delivered under a firm-fixed price contract your tolerances will be very strict, so you may want to set your estimate at the 95th percentile or higher (in your table your estimate may be more than 80 minutes).
  • For projects that are low risk and of low complexity your tolerances will be much less strict, so you may set your estimate at the 65th percentile and go from there.

This relates to the issue of accuracy ("how close did I get to my estimate") vs precision ("what is the expected error in my estimate"). I think a strong argument can be made that at the very start of a project the precision of an estimate is probably measurable in weeks, which should feed into how strict your tolerances are.

  • So is it about your target being accurate / precise or is about your ability to predict the future? Most of the variability in performance we experience is random and uncontrollable. Do you disagree? – David Espina Aug 27 '12 at 14:49
  • The issues of variability of performance and precision of estimates are related. My point is that accuracy and precision are different and having excessive precision in one's estimates may not add value. Trivial example - For sales tax of 12% on a purchase of $1.99 a precision beyond two decimal places has no value, so final price is $1.23 rather than the (precise and "correct") $1.2288. Likewise, if my estimates have an error of weeks why give a target date rather than a target week/month? – Doug B Aug 27 '12 at 18:03
1

Funny, I use exactly the same example of commute time in my training classes. The point I try to explain is that a (single point) estimate can never be accurate (nor precise, which is a synonym I believe), as it would imply a 100% chance of happening. The probability distribution of a well known task (commute) clarifies the variability. Hence the advise to use ranges (Most Likely - Worst Case) and carefully decide what risk you wish to take, how much buffer or reserve you want to take into account etc. (I can explain this further, but the question is about 'accuracy' I believe).

I don't see the point in calling a range estimate 'accurate'; how it was calculated is more important, and how much risk is implied (Worst case only 20% higher or a 100%)?

I'm a bit worried about the notion of "data expires" discussed by Zsolt: taking out the 'road block' data from the commute time example, would imply no other road blocks are possible on your way to work. In light of risk management, I would certainly leave it in (unless you have the time / and money to test drive and measure every possible alternative route for getting to work).

  • 1
    Accurate and precise are different, not synonyms. Accurate is how close it is to the actual, while precise is how certain you are. Saying "It will take me 1 minute" is precise, even if it takes you 20 minutes. Saying that it will take you 0.5 hours is less precise, but would be more accurate if it takes you 20 minutes. – Thomas Owens Aug 27 '12 at 18:59
  • Might be, I'm not an English native speaker. But that would mean my point about accuracy is still valid, and a 'range' is precise? By the way, why would 30 minutes be less precise if you're dead certain about it (even if reality proves you otherwise; this sometimes happens with estimates ...) – Stephan Aug 27 '12 at 19:10
  • That's right - the majority of the rest of your post holds (aside from the statement that a single point estimate can never be accurate - one can be both accurate and precise). However, estimates are rarely ever certain and what single-point estimates don't do is capture the uncertainty. As an aside, I think the "accuracy"/"precision" thing typically exists in the sciences and engineering and not in common English usage, where they are treated as being the same thing. – Thomas Owens Aug 27 '12 at 19:13
  • Just saw the edit to your comment. The range is only one method of capturing the uncertainty. There are multiple ways of capturing uncertainty - giving a probability of certainty, a range and that range's confidence interval, a single point plus a list of the top factors that might cause deviation, best/most likely/worst case. – Thomas Owens Aug 27 '12 at 19:15
  • 1
    @jmort253 In Software Estimation, McConnell uses precision the way I do, to reflect the units on the estimate. An estimate given in day is more precise than one given in month. He also uses precision to reflect the usage of significant digits when the units of the estimate are the same, saying that an estimate of $1000 is less precise than $975.00. If you have the book, check out pages 51-52. To me, repeatability only matters for both accuracy and precision over multiple projects (can you get a small unit close to the actual for every project) or for parametric estimates (does the model hold). – Thomas Owens Aug 28 '12 at 11:16
1

@Aclear has it right - an "accurate estimate" is one that falls within the acceptable margin of error for that particular instance.

If I say the work will take 5 days and it takes 5.5, then within the overall scope of the work that may be an accurate estimate. A) there was no way to determine beforehand 'exactly' how long it was going to take, so 1/2 day on 5 days is probably pretty good.

On the other hand, if I estimate it will take 20 minutes to drive my kids to school and I leave with 20 minutes until school starts, if it takes 25 minutes and they arrive 5 minutes late then my estimate was not accurate. It was outside the acceptable margin of error.

In reality, an estimate is just that, and estimate. Whether it's an accurate one or not really depends on how accurate I NEED to be.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.