5

Question

Is it correct to assume that the criticality of a defect can change along with the project's progress? If so, when is this acceptable?

Background

Criticality is one of the indexes we've used to use to define what can fit into a release or not. There were cases when the criticality has changed when users were not willing to add specific functionality because at that point in time the functionality was not priority. When the functionality became priority (and other critical defects were fixed), we have had a second round of criticality reviews moving entries into the queue for prioritization, since now they are considered critical.

My impression is that if this functionality is critical for business operation, then it should have the same level since day one. Once the defect is clear, the criticality isn't expected to change. On the other hand, if the defect wasn't clear enough, it's expected to have the criticality changed as the project gains clarity about it.

| improve this question | |
  • This is a great question! I tried to retag this to reflect that it seems to be largely about managing stakeholder estimates of what's critical, rather than about testing or QA, but I wasn't sure that any of our existing tags were perfect fits for the underlying question. Maybe someone else can do even better with the retagging. – Todd A. Jacobs♦ Nov 28 '14 at 16:47
  • Thanks, CG. For my surprise, I also had a hard time picking up the tags for this question... – Tiago Cardoso♦ Nov 29 '14 at 9:53
3

Super-Short Summary

The criticality of an issue is contextual, and can change over time as well as due to framing or extrinsic factors. I describe some silly but illustrative examples below, as well as a couple of more practical considerations of "value."

When is "Critical Bug" Really Critical?

Imagine you have a new phone app, which has a serious bug: every time the user taps the screen inside the app, the user is electrocuted with 200,000 volts of electricity. The organization has labeled this bug as "critical" because electrocuting customers creates bad PR, and reduces the rate of future in-app purchases from customers who are...well, dead.

But what if circumstances change? Can this impact the criticality of the bug from a business point of view?

The Same Bug, Present But No Longer Critical

Reinterpretation as a Feature

Perhaps your marketing department discovers that there's a huge, untapped market in Transylvania by selling the app to the laboratories of a Dr. Frankenstein. You are now selling 1,500 units per week to the good doctor, and your sales are therefore stable. This bug is now considered a "feature" for some sub-set of your customers, so it is re-labeled as a wish-list item to make high-voltage discharges an optional feature rather than an automatic one.

Compensating Controls

Alternatively, perhaps each copy of the app now sells with an absolutely-free grounding rod and a non-conducting pair of gloves. The PR department says every existing or future customer will get this safety gear "out of an overabundance of concern for customer safety, even though we have zero post-discharge complaints from customers who have personally experienced our market-leading voltage enhancement feature."

Same Issue, Different Criticality

In either case, the issue is deemed less critical because of changed circumstances, a different context, or even plain old marketing spin. The issue is no less real, nor has it gone away. Instead, the value proposition of the issue has changed—perhaps sufficiently that it isn't even considered a bug any longer—due to external or internal factors.

The Value of Features (and Bugs) Change with Time and Lessons Learned

Even aside from the contrived example above, a bug that seems like a show-stopper early on in a project may not remain on the critical path or provide the same level of value later on in the same project. Value changes over time based on a wide variety of factors, and the severity or urgency of any given bug will also change as the value proposition of its related functionality changes, too.

Consider a less silly example where a system has usernames, and the bug is that usernames aren't unique within the system. Is this bug still "critical" if the system moves to using email addresses as account identifiers? What if the application switches to an open, CB-style chat application that doesn't need to retain account information at all?

Bugs are only bugs insofar as they prevent a system from performing a desired function. In other words, a bug subtracts value from a product. In an agile project, the "bugginess" of any issue should change along with the project's evolving notion of value. Your mileage with the WaterFailure™ model will of course vary.

| improve this answer | |
  • Hehe, I really loved the examples, CG! I agree that the "bugginess" of any issue should change along with the project's evolving notion of value, but what about a situation where people assess a defect today as 'low' even knowing that in 3 months from now the defect will become critical? That's the case that, for me, seems like the cheating the system Marv mentioned on his answer. – Tiago Cardoso♦ Nov 29 '14 at 10:03
  • @TiagoCardoso: That depends on why the two classifications are so different. If the bug only seriously affects a feature that isn't on the development horizon for the next few months, then it is entirely logical to classify it as 'minor' now and as 'critical' when work on that feature is about to start. – Bart van Ingen Schenau Nov 29 '14 at 15:44
  • @TiagoCardoso Because bug classification is a value judgment, changing the classification of a bug is quite common. Gaming the system isn't wise, but I would simply treat deliberate mislabeling of bug severity or urgency as a valid topic for a Sprint Retrospective or similar inspect-and-adapt meeting. If bug classification changes in a transparent fashion, great! If it's used a way of sweeping issues under the rug, though, then it's the PM's job to haul the issue out in the open to create greater project transparency and issue visibility. – Todd A. Jacobs♦ Nov 29 '14 at 15:53
  • 1
    create greater project transparency and issue visibility then, the only way to avoid gamification is setting up some ground rules on how (and in my case, more specifically when) defect classification review will take place. Thanks for this great insight! – Tiago Cardoso♦ Nov 30 '14 at 15:07
  • For clarity, I wasn't suggesting gaming the system to achieve acceptance, in case that was inferred. Merely that I often find that issues users define as super-critical, must-haves, often become less critical when they are reviewed at the end of the UAT cycles and found to be the only thing preventing a product from shipping. I often find the users agreeing that, actually, they can live with this or that defect after all for a short period and nail it in the next release. It is a matter of shifting perspectives :) – Marv Mills Dec 1 '14 at 10:23
8

I love defect management, it throws up some wonderful logic problems :)

What actually is Criticality? If you could define that then you would know the answer.

I believe it is the combination of Severity and Priority, both of which should always be defined independently in defect management. The severity of a defect, i.e. the impact it has on the suitability of the system, should not change other than if external factors cause it to become more or less of an impact. Whether or not it is to be fixed and when dictates the priority. The priority can change depending on the needs of the project, the needs of the stakeholders/user, or a million other variables.

So therefore the Criticality of a defect is infinitely changeable :)

I often structure my Acceptance Criteria to specify the levels of known defects to be allowed into the deliverable without the project being deemed a failure. Typically this will say something like "No Priority 1s, no more than two Priority 2s and 3s and any number of Priority 4s". Note that this is not graded on Severity because a very Severe bug might only occur once in a blue moon whilst a less severe defect might cause daily operational problems- Therefore fixing the lower severity issue becomes the top Priority.

This latter approach also has the beneficial effect that you can manage a deliverable through Acceptance Criteria merely by reclassifying the priority of outstanding issues- This is often welcomed by the project team and the UAT teams alike, even though it seems like cheating! :)

| improve this answer | |
  • "even though it seems like cheating!": here you have my underlying question - once the top priority is fixed, another cycle of defect review will take place and another issue will become top priority... is it realistic and PM-acceptable? – Tiago Cardoso♦ Nov 29 '14 at 9:57
  • 1
    This is extremely common and, in my view, absolutely part of the defect management cycles. The priorities should be set by the users with assistance from the PM and BAs. As long as that happens why shouldn't their priorities change as the situation changes? In the end it is the users (via their sponsors and stakeholders of course) that we are serving and for the project to be successful the users need to accept the deliverable, so meeting their needs in terms of defect priority is at the core of the PM's duty I believe. – Marv Mills Nov 29 '14 at 10:26
  • The distinction between priority and severity is very important. A defect that takes a low priority function offline can have a very low priority, thus effecting its overall critical-ness. – Andrew Clear Nov 29 '14 at 20:06
2

Defect is a gap between actual system behaviour and expected by user. Defect criticality is a size of this gap. Either system behaviour or user expectations change - so does defect criticality.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.