Should defect criticality change based on user needs?

Question

Question

Is it correct to assume that the criticality of a defect can change along with the project's progress? If so, when is this acceptable?

Background

Criticality is one of the indexes we've used to use to define what can fit into a release or not. There were cases when the criticality has changed when users were not willing to add specific functionality because at that point in time the functionality was not priority. When the functionality became priority (and other critical defects were fixed), we have had a second round of criticality reviews moving entries into the queue for prioritization, since now they are considered critical.

My impression is that if this functionality is critical for business operation, then it should have the same level since day one. Once the defect is clear, the criticality isn't expected to change. On the other hand, if the defect wasn't clear enough, it's expected to have the criticality changed as the project gains clarity about it.

Answer 1

Super-Short Summary

The criticality of an issue is contextual, and can change over time as well as due to framing or extrinsic factors. I describe some silly but illustrative examples below, as well as a couple of more practical considerations of "value."

When is "Critical Bug" Really Critical?

Imagine you have a new phone app, which has a serious bug: every time the user taps the screen inside the app, the user is electrocuted with 200,000 volts of electricity. The organization has labeled this bug as "critical" because electrocuting customers creates bad PR, and reduces the rate of future in-app purchases from customers who are...well, dead.

But what if circumstances change? Can this impact the criticality of the bug from a business point of view?

The Same Bug, Present But No Longer Critical

Reinterpretation as a Feature

Perhaps your marketing department discovers that there's a huge, untapped market in Transylvania by selling the app to the laboratories of a Dr. Frankenstein. You are now selling 1,500 units per week to the good doctor, and your sales are therefore stable. This bug is now considered a "feature" for some sub-set of your customers, so it is re-labeled as a wish-list item to make high-voltage discharges an optional feature rather than an automatic one.

Compensating Controls

Alternatively, perhaps each copy of the app now sells with an absolutely-free grounding rod and a non-conducting pair of gloves. The PR department says every existing or future customer will get this safety gear "out of an overabundance of concern for customer safety, even though we have zero post-discharge complaints from customers who have personally experienced our market-leading voltage enhancement feature."

Same Issue, Different Criticality

In either case, the issue is deemed less critical because of changed circumstances, a different context, or even plain old marketing spin. The issue is no less real, nor has it gone away. Instead, the value proposition of the issue has changed—perhaps sufficiently that it isn't even considered a bug any longer—due to external or internal factors.

The Value of Features (and Bugs) Change with Time and Lessons Learned

Even aside from the contrived example above, a bug that seems like a show-stopper early on in a project may not remain on the critical path or provide the same level of value later on in the same project. Value changes over time based on a wide variety of factors, and the severity or urgency of any given bug will also change as the value proposition of its related functionality changes, too.

Consider a less silly example where a system has usernames, and the bug is that usernames aren't unique within the system. Is this bug still "critical" if the system moves to using email addresses as account identifiers? What if the application switches to an open, CB-style chat application that doesn't need to retain account information at all?

Bugs are only bugs insofar as they prevent a system from performing a desired function. In other words, a bug subtracts value from a product. In an agile project, the "bugginess" of any issue should change along with the project's evolving notion of value. Your mileage with the WaterFailure™ model will of course vary.

Answer 2

I love defect management, it throws up some wonderful logic problems :)

What actually is Criticality? If you could define that then you would know the answer.

I believe it is the combination of Severity and Priority, both of which should always be defined independently in defect management. The severity of a defect, i.e. the impact it has on the suitability of the system, should not change other than if external factors cause it to become more or less of an impact. Whether or not it is to be fixed and when dictates the priority. The priority can change depending on the needs of the project, the needs of the stakeholders/user, or a million other variables.

So therefore the Criticality of a defect is infinitely changeable :)

I often structure my Acceptance Criteria to specify the levels of known defects to be allowed into the deliverable without the project being deemed a failure. Typically this will say something like "No Priority 1s, no more than two Priority 2s and 3s and any number of Priority 4s". Note that this is not graded on Severity because a very Severe bug might only occur once in a blue moon whilst a less severe defect might cause daily operational problems- Therefore fixing the lower severity issue becomes the top Priority.

This latter approach also has the beneficial effect that you can manage a deliverable through Acceptance Criteria merely by reclassifying the priority of outstanding issues- This is often welcomed by the project team and the UAT teams alike, even though it seems like cheating! :)

Answer 3

Defect is a gap between actual system behaviour and expected by user. Defect criticality is a size of this gap. Either system behaviour or user expectations change - so does defect criticality.