So my boss put forth a 'research and find out question' to me:

  • Are there ANY (absolutely any, no matter the dumbest, but any) metrics for requirements management? Eg. Requirements completeness, consistency and productivity (being the most difficult to nail down i.e. what is meant by productivity).

Similar question with respect to design and architecture level metrics. I.e., do any exist and what are they if they do.

It's kinda fuzzy right now but I wanted to know what are the metrics that you have commonly used at these stages (if at all) and what have you found to be worthwhile. The notion is that most companies can (and do) come up with proprietary metrics to measure stuff related to the requirements/design/architecture phases but are there any 'commonly applicable' ones?

Note: I'm not looking for code related metrics like cyclomatic complexity or bugs/100 LOC or function points

I'm aware of the product backlog in the agile community along with release burndown release burndown charts tend to give you a good idea of things but they don't work as well in a non-story-point estimation environment, IMO. They are good proxies to measure the aforementioned requirements consistency, completeness and productivity aspects but not sure about how well can they cater to that.

I'm well aware of it being a hard problem, I'm not looking for a solution but mere opinion/view points on what have you used or seen/heard being used in this regard. Research paper references more than welcome too :)

  • Not a duplicate, but similar to What are good metrics for an SRS? – Adam Wuerl Jul 8 '11 at 16:02
  • Yup! Had looked at that. Most texts say that you should "track" such metrics, my question is "what metrics" have been able to quantify those ideas concretely and what has been the experience of using them. Eg. It's good for requirements to not be ambiguous, but is there any (no matter how obscure, less used, dumb) metric that helps quantify ambiguity?? – PhD Jul 8 '11 at 16:04
  • The best and only surefire way I know to ensure a requirement is unambiguous is to document the verification methodology. Not just the method (e.g. analysis, text, inspection, etc.) but a detailed description of exactly what you're going to do and objective pass/fail criteria. That way the meaning of the requirement is clear. Once the system has been put through the verification methodology and passed, that's what it means to satisfy the requirement. – Adam Wuerl Jul 8 '11 at 16:10

In terms of metrics for a set of requirements:

  • number of requirements is a good indication of system complexity (it may seem silly at first as not all requirements are the same "size", but in aggregate it seems to work no worse than counting lines of code does for software complexity)
  • number of TBXs both in absolute and percentage of requirements that are not yet complete. It may also help to track the different flavors, as a TBD(etermined) is less mature than a TBR(esolved/reviewed) or a TBS(supplied)
  • number of requirements with verification methods and methodologies, as requirements that no one has thought about how to verify probably aren't that good yet
  • number of requirements flowed-down to lower-level specifications or similarly the number of childless requirements in top-level specs or orphans in lower-level specs

If you're talking about measuring the efficiency of your requirements management processes:

  • hours charged per change request released
  • verification burn-down plan vs. actual progress
  • percentage of the specifications in the specification tree that are released and under configuration management
| improve this answer | |
  • Nice Set! I had suggested the number of original captured requirements to number made it in final deliverables and so on. But the question was "is there anything in literature or should we come up with our own" :) And that's the one making me scratch my head (and hopefully the community's too :) – PhD Jul 8 '11 at 16:33
  • 1
    If there is a body of literature on requirements metrics then I'm not aware if it, which is entirely possible. But most big companies I know of have captured and distilled their institutional knowledge into internal processes. That is, I would not be surprised if you failed to find anything in literature and decided your best bet was to think if your own and capture them for future use like several other companies have done. – Adam Wuerl Jul 9 '11 at 19:38
  • Absolutely! That's "exactly" what I told my boss :) – PhD Jul 10 '11 at 5:03

Although it's not exclusively related to requirements engineering or design and architecture phases, defect removal effectiveness can be used to find out how effective your requirements engineering and design activities, along with all any other activities throughout the lifecycle are.

The idea is that you create a table that shows where defects were found, where they were fixed, and where they came from. Using this information, you can find out how many requirements defects were found in requirements, found in design, or slipped all the way through to field deployment. You can track this per iteration (since every iteration will deal with requirements engineering through deployment) as well as per project. However, you need to complete a few iterations to have sufficient information as to the effectiveness of your practices at each phase and make corrections.

Other measurements and metrics would be related to rates of change of requirements, such as added, removed, and modified requirements. Also, relating requirements to implementation time or test coverage might also be suitable. I think that Adam Wuerl's answer discusses this pretty well, though.

| improve this answer | |
  • Yup! We do that very well currently! That is exactly (and probably only) types of metrics we track...I am more keen on finding out other more direct ones as I mentioned in the OP – PhD Jul 10 '11 at 17:50
  • 1
    @Nupul The only other thing that I've found that might be interesting would be correlating customer satisfaction to requirements. There is probably some kind of a link between "how good the requirements are" and "how happy the customers and users are with the software". Unfortunately, I couldn't find any strong research in this direction. Perhaps looking at research and studies on TQM might yield something in this direction, I'm actually a little surprised at the lack of easy to find and/or well-known work toward downstream activities and software quality. – Thomas Owens Jul 10 '11 at 18:02
  • Totally agree on the difficulty of finding... – PhD Jul 10 '11 at 18:41

Revic 9.2 cost estimation model also estimates the requirements and review stages. Note, though, that it's based on lines of code, so it's a ballpark estimate. Plus it's a U.S. Air Force model, so it's aimed at military projects.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.