What are acceptable quality levels for various kinds of software?

Question

Is there any statistics or at least some industry consensus on what are commonly acceptable quality levels for various kinds of software?

I am certainly not talking about space shuttles or life supporting machines where the answer is more or less obvious. Rather, I am talking about such things as casual games, enterprise apps, e-commerce websites etc.

Again, it is pretty obvious that even with these types of apps, defects causing loss of money or similar severe consequences are unacceptable. But what about crashes, intermittent malfunction, and defects having an awkward, but still a workaround?

Please bear with me here, as I've seen a number of internal apps that are slow, buggy, ugly as hell, but are nevertheless successfully helping to run businesses and are actually used despite all their deficiencies.

Moreover, I've seen cases where customers were intentionally not willing to pay for good quality: absence of major/critical defects was all they needed.

Of course it always helps to ask for a specific customer's expectations, but having some reference points/examples would greatly facilitate such a conversation as well as help properly plan the amount of prevention and appraisal efforts.

Answer 1

This is a decision that should be made early on in the planning process. "What bugs are we willing to ship with?" Your question clarifier ("we're not talking about human-rated, life critical software") is something that cannot be repeated too many times when talking with QA or Management, who might have unrealistic expectations.

Answer 2

I am not a SW guy so I am putting my answer in from a generic perspective. Like Scott indicated, spec limits are/should be predefined. This is true for even life critical products; the tolerance variation would be less, but there is still variation, e.g., defects.

Where no predefined spec was indicated, it comes down to risk and decision analysis. The cost of repair compared to the expected economic impact (probability x financial impact) going at risk. The cheaper one wins. Next time you board an aircraft, think about that!

Answer 3

I worked for a company a few years ago as the PM who was required to implement a whole set of new software systems for retail stores and a distribution warehouse, and lead the implementation of new business processes to allow the clever software to do what it had to do. The project was a massive success in business terms, despite problems with the software that impacted on its usability in some areas, gaps in functionality in others, and very limited management information provided as standard reports.

What made the whole thing work was the attitude of the software company, who guaranteed that the underlying functionality would be completely OK; they promised to fix major usability problems as a priority over developing new functionality; they would develop new functionality ahead of cosmetic changes, and they refused to develop a reporting capability, saying from day 1 (before we bought the system) that it was our data: we had to develop our own reports (but they did give us tools to help).

Astonishingly (in hindsight) they convinced us to buy their system, and provided a great deal of support, but they also provided excellent liaison and built lasting relationships with our technical guys and, even more importantly, with the system users.

That's a long way of saying that if the data is right, you can put up with a lot of problems as long as you feel that the developers care about you. I suppose that they were using a sort of agile process, although neither they nor I ever thought of it in that way, but their developers were quick, accurate, and understood the business.

Despite that amazingly good experience, I would be nervous about deliberately going down that same road again, as I think I just struck lucky, but the principles of good data, OK processes, and the ability to quickly change the user interface probably define my philosophy toward development priorities.

Not sure whether that answers your question, but it is a practical example of how imperfect systems, developed in the right way and with appropriate levels of care in the key areas, can be massively successful. We certainly would not have turned the company around if we had waited for perfection.

Answer 4

Capers Jones has been publishing information about defect density for various kinds of software for years. Unfortunately, you have to pay to get their reports, but there are a few folks who have quoted the summary numbers. Here is one from Watts Humphrey's article fittingly entitled "Defective Software Works".

According to that data, the least disciplined class of development tends to produce software that has on average 10 defects per 1000 lines of code (KLOC). The most disciplined 1 defect per KLOC on average but that varies wildly. I've seen safety critical systems shipped and never have a single customer reported defect.

There are a lot of different ways to think about defects. You can categorize them into functionality shortcomings, UI quirks, with workaround versus without, data corruption Y/N?, etc. However, I tend to try to boil it all down to economics. What's the risk that buggy software will hurt sales? The answer will depend largely on who the decision maker is. If it's a $5 app downloaded from an App Store, then a trouble free experience is key. If you are building enterprise software where the decision maker is not an actual user, then you'll have a different model.

Using this economic thinking, one of the key factors to keeping the cost of defects down is to remove them as early as possible. The traditional model says that removing a requirements defect during a requirements elicitation activity is 1/10 the cost of doing so during design, and 1/100 as expensive as removing it during implementation, and 1/1000 as expensive as fixing it after the software has shipped. This argues for requirements review, design review, code review, etc. While there are still appropriate levels of review, the Agile mindset has a different way of thinking about it.

The Agile argument is that if you focus on making your software easily evolvable, you can push it out early and get better feedback from actual users than you do from requirements/design/code reviewers. Quantity is its own form of quality. It's cheaper and you end up with a better result if you get something out there and replace it several times than it is to think long and hard about how to make it perfect the first time. Let the users tell you what is important and unimportant.