What software quality KPIs do you use?

Question

I need to define the KPIs which will help me to say if software coming out from the project is good or bad. I mean does it fulfill quality KPIs which were targeted or not. The question of course is what KPIs can be defined for software development project. What quality KPIs do you use in your software development projects? How you interpret them? Thank you all in advance for help!

For Change management in my project we use MS Team Foundation Server 2010. All processes are very loosely based on CMMI. The project is similar to SAP. We have a modules which are the core of the system, lower levels are common for smaller parts of stakeholders or customize. We are working in a mode of support of current released system versions (fixing bugs, adding users etc.) and also creating new futures. The project is web application.

We work for internal client.

Answer 1

Actually, I'm using only two KPIs:

• number of users/customers: as a goal, we define how many users or customers we would like to have during a certain period. For example, we want 30,000 new customers in this year

• the time we need to deliver a new feature: in other words the lead time. We check our actual lead time and see check whether this time is enough to stay on the market. For example, we want to reduce our current 10-day long lead time to 9 days in this year, because the competition became faster then us

As you can see these KPIs/goals are about the trends: "more" customers and "less" time spent. Additionally, the set numbers are not achievable with our current way of working, so the organization has to work and improve in order to achieve them.

We have automated tools to get the data above and we regularly check on our indicators whether we are on the right track or not. If not, we change our way of working, try out new stuff etc.

I no longer use KPIs like number of defects, or lines of codes, or test coverage, because

• they don't help to sell more product or stay on the market
• they are hard to set on a whole organization (e.g. sales cannot do anything about coverage)
• they narrow down what employees shall do, and they won't bring their creative ideas into the organization

I give you an example. In my friend's organization one of the main KPIs is the number of defects in the system. As far as I remember, the goal is to have maximum 10 defects. They keep this number, even if it is not necessary. Their customer can live with 20 or 30 defects, he wants to have a system with fast reaction time, but the developers spend time on defect fixing and not on making the system more faster. There is chance that they'll lose this customer over this. If they had the KPI "number of customers", each team could set its own goal/action to see what to do in favor of this KPI and keep the customer.

Answer 2

There are several KPIs -what we call a quality model- you can use to measure a software product quality. One approach, and a good starting point is the standard ISO 9126 that recommend to measure the internal quality, the external quality and the quality in use of your product and based on the different metrics you can gather infer the different aspects of quality: maintainability, functionality, usability, portability, efficiency and reliability of your software.

There are commercial and non-commercial products that can help you calculate the aspects using static code analysis and integration with testing platforms for example.

Some of the underlying metrics include level of compliance to predefined coding rules and best practices, intrinsic metrics like cyclomatic complexity, size metrics like lines of code or function points, etc. The previously mentioned number of users and time to deploy new features alone, I don't think measure the real quality of your product. At most they can help to calculate some of the aspects, like usability, but the view they give is very limited. You need a more holistic view and for that you need more data.

I want to mention that measuring the real quality of your product throughout the development life-cycle pays off. You will have less defects in production and your maintenance cycles will be shorter. This is proven.

Answer 3

For progress reporting I'm a big fan of Earned Value. This is the most honest method for tracking schedule and budget, but may not reach the Quality metric you're looking for.

Moving away from Earned Value, for custom and enterprise systems, I try to work back from the business case. If the business case is, "Save 15 minutes per user per day" then I track how much of this I've demonstrated completing. "We've knocked off 5 minutes, we've achieved 1/3 of the business case" (This may sound like Earned Value, but it's actually different)

In between the business case and the schedule are other quality metrics. I like to know where defects are being found by release, sorted by severity. Are the end users finding them? Are we getting them in system test? Unit test? Are they centered around certain parts of the application? The metrics alone aren't perfect, but they can help you look.