Caution or Waste?

Question

Are there any strict definitions regarding the difference between caution/security/safety and waste? For example, consider the following scenarios:

1:

"We just finished this. We'd better test it to make sure it actually works."

2:

"We're going live in a week. We'd better re-test this thing to make sure it still works."

3:

"We're going live tomorrow. We'd better re-test this thing."

"I don't think we've actually changed anything, though."

"Are you 100% sure?"

"99%."

"Then we'd better test it again, to be safe."

4:

"Okay, it passed. Are we done?"

"No, run it one more time to be sure."

"...Why?"

"It's possible you made a mistake when testing it. Run it five times, to be completely certain."

Which of these (if any) would be considered as waste? Which as just being cautious/safe? Is there a formal metric for determining such, or is it opinion-based?

Answer 1

I don't believe that the two are mutually exclusive. Something can be wasteful to varying degrees and be cautious and be valuable. While not completely subjective, I'm not aware of a particular measurement to indicate that something is wasteful. One definition I saw for waste is: “useless consumption or expenditure; use without adequate return.” In other words, anything that does not create value is waste.

Let's take your third example. Is there waste in doing a complete retest to go from 99% sure to 100% sure? I'm sure there is. How did we get to 99% without getting to 100%? I can't fathom that any real-life situation could get you here without being in some way wasteful.

On the other hand, if the thing going live runs the New York stock exchange, it may be appropriately cautious to put in the extra effort to get to 100%. That doesn't mean that it isn't wasteful and that you shouldn't strive to drive out that waste, it just means that the most appropriate level of caution would be to accept that waste for this current action and look to solve the problem moving forward.

Answer 2

Assuming you have an established QC/testing method and reliable and valid pass or fail criteria, and you conducted the test, anything beyond that is wasteful.

That's not to say you won't find issues if you conducted the additional testing but the cost of finding them as compared to the find and their impacts no longer make financial sense.

Quality controls, and any type of risk mitigation, cost resources as well as the assumption of secondary risks. Again, assuming a well constructed testing method and pass criteria, any additional mitigation done in some sort of ad hoc manner (which is what your OP implies), chasing whatever fears you may have, will cost you resources and secondary risks that will exceed the probabilistic impact value of what you may find. This is a quintessential definition of waste. Remember EMV and decision analysis?

The testing method and criteria are supposed to equal the level of risk mitigation cost and degree of risk you are willing to assume for that project at that time.

Answer 3

Your first 3 cases are nowhere near wasteful; I'd consider them as SOP.

1. Testing early and often is the only way to ensure you don't have a bucket full of trouble when you're almost done; some fixes may require rethinking of other parts.

2. Testing one last time before release ensures that you are releasing the version you plan on, and nothing has accidentally been changed.

3. Testing once again in a pre-release staging environment and then immediately after going live is sensible and I've always insisted on it.

Even your 4th case - test again and again - is almost SOP, as some things do break only on subsequent runs or when run quickly one after another. This is usually part of a good test plan.