I have mostly worked on projects which followed Waterfall model and we had in place lot of checklists and audits during the SDLC which ensured that we were able to meet the quality goals.

Now I work at an organization where majority of the projects are using either Scrum or Kanban. Lot of times requirements/stories are given by the users/client directly. There is very little prioritization that we do with the client. The project size varies from 2 developers to 20 developers. The estimates given by the developers for their stories sometimes miss out the unit testing, code review/rework effort. I have educated them and now they have started estimating it correctly.

What all processes can I introduce so that the quality parameters like defect injection rate, code review effectiveness, software design effectiveness can be in control? I think lot of the developers confuse Agile/iterative model with short cut development. So educating them is first thing that I am doing.

I will like to ask the experts here if there are any thumb rules which I can use or if there are any best practices which I can implement so that all projects follow similar process, projects follow more predictable path, the number of stories delivered is higher and quality parameters can be achieved.

  • Yes! Many developers in my experience also confuse agile as meaning it's okay to take shortcuts or be sloppy. That is not what agile is about. :)
    – jmort253
    Jun 6, 2015 at 7:44

1 Answer 1


I think that in the agile methods, you're probably going to want a different set of quality metrics to track against.

In an agile project, especially with regards to the design and code and test activities, it's highly iterative and there may not be a clear separation of these activities, so phase-based metrics (measuring effectiveness of design and code reviews, for example) become hard. In addition, since testing typically happens concurrently and continuously, I don't think you'll get typical defect arrival curves.

I think you'll want to keep the Agile Manifesto in mind when developing new metrics to track quality. Specifically, the Agile Manifesto favors "individuals and interactions" and "customer collaboration". You'll also want to keep in mind the Principles Behind the Agile Manifesto, especially the first one: "our highest priority is to satisfy he customer through early and continuous delivery of valuable software".

Your first metric should be one of customer satisfaction. A good starting measurement would be defects reported by customers and end users. If you're tracking reported severity or priority, this could be part of that measurement. You can also track the number of iterations to resolve defects (likely based on priority and/or severity). This type of metric relates to defect turnaround time (time between reporting and verification) or defect response time (verified defects being resolved by the development team). You could even build up goals around these data points to improving your quality activities to the point of 0 "significant" (based on your definition of "significant") issues reported by customers in releases.

A good agile metric for customer satisfaction would possibly be the number or percentage of user stories that were accepted at the end of the sprint. Ideally, you should finish every planned story and every finished story would be accepted. That's not always the case, though.

Since you're also using Kanban, tracking a severity distribution (something that ranges from cosmetic defects to blocking issues) and priority distribution (something that ranges from low to high) along with the swimlane that the work item is in may yield interesting data. For example, once a developer says that a story or task is done with development, there should be no blocking or high priority issues found in it. Tracking that a large number of items are making it into acceptance testing with defects over a certain severity or priority could trigger actions.

A little closer to the development team, a good metric may be automated test coverage. The agile methods tend to favor continuous integration, and automated unit, integration, system, and regression tests are well supported in this type of environment. Tracking coverage, especially in new code, may be useful. Of course, test coverage is not a quality metric (see also: How to Misuse Code Coverage) since it doesn't say how good your tests are, but you can use it with other defect reports found during test or post-release activities to know how good you are. You can use it as a quick check to ensure that some tests are being written for new code, though, and that as issues are found, tests are being written for inclusion into test suites to prove the issue and then that the issue has been resolved.

From a requirements perspective, the stories brought into the sprint should be relatively well understood and what you will be working on. Although Agile recognizes the fact that requirements can and do change and you should embrace change, you usually don't want major changes to the stories that are part of the current iteration. You should probably want to measure the volatility of stories that you've brought into the sprint - once they go through the grooming process and are accepted by the team, track the number of changes to some aspect of the story (including stories being added or removed). The addition or removal of a story is visible on a burn-down or burn-up chart, but changes to a story may not be reflected there.

As an aside on requirements, I am concerned when you say that your users and client directly provide requirements and stories and there is very little prioritization done with the client. If you don't have an on-site user or client to act as the Product Owner, there should be a person on your team that can act in this capacity and has a solid understanding of the needs and desires of the users and client. The Product Owner should be responsible for prioritizing stories and providing clear acceptance criteria. Having prioritized stories and clear acceptance criteria will be visible in measures such as requirements volatility and defects discovered following a release.

Although it does add overhead, if you're working on a contract that requires time tracking already, consider tracking time against certain tasks. It depends on how you break down your work into tasks, but if you have to track hours worked, you can track cost of quality. Things to look for are how much time you're spending writing and executing tests (manual and automated), peer reviews of designs or code, and time reworking designs or code because of failed tests. Of course, be careful that this doesn't add a lot of overhead - it may not be suitable for projects that don't require time tracking.

From a process quality perspective, examine how good your estimates are. It's a little harder with story points on a per-story basis, but you can track your overall velocity across sprints. After a few iterations, you should be able to consistently tell how many story points you can complete in a single sprint and then hit that number (maybe with some minor variations). Your burn-down charts and velocity tracking can be looked at over time to make sure you're properly scoping and executing your work items.

I do have one last word of caution. I would recommend embracing the Lean Software Development principles. Make sure that any measurements and metrics that you are collecting are actually adding value to the customer, the team, or the organization. When your metrics or indicators show a problem, empower the team to make decisions on how to go forward and make corrective actions.

  • Thanks Thomas for your answers. When I mentioned users give the stories, i forgot to mention that users/client are the product owners and are aware of their needs and priorities. We are putting a lot of focus on CICD and TDD currently. I am also doing pareto charts, scatter diagrams for the defects analysis. Number of iterations to fix a bug is something that I dont track. I am going to implement lot of things you mentioned. thanks again
    – ViSu
    Jun 3, 2015 at 11:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.