Method for assessing risk in a release

Question

I have been tasked to develop a metrics program specifically designed to look at how risky a release is from a code base volatility perspective.

The director requesting this information is working from the assumption that more change to the overall code base later in the release cycle (percentage wise) increases risk. It seems a reasonable assumption but I would like to numerically justify it before we start reporting it to the development teams on a regular basis.

My rough (very rough) plan at this point centers around a few data points.

1. Quality data: Review production defects/incidents and tie them to release X (where possible)

2. Quality data 2: Review pre-production, test cycle defects tied to release X

3. Change/Quantity data: Review code volatility numbers either at the file level or by # of changes per application. I am having trouble locking down the best quantity measurement to use...lines of code seems to detailed and reviewing by change list seems too broad so I am thinking somewhere in-between.

Questions:

• Does this plan create a system that can help evaluate risk at the release level?
• Are there other measures I have not considered that should be included?

Answer 1

TL;DR; The first two metrics aren't bad, but the first is hindsight and the second organized information you already know. The third might line up sometimes, but will almost certainly get you in trouble.

The three important questions when trying to establish any metrics should be:

1) What exactly does this metric tell me?

2) What actions can I take based on this metric?

3) What behavior will this metric cause in my team?

While your first metric may be difficult to do, it's a great metric for looking back at your releases. Your second is also a good one, but it tells you what you already know - how many defects were found already in the changes. If you're interested in code risk, one thing that might be good is to see how many of those defects impact something that wasn't supposed to change. That could be an indicator of volatility problems in a release (though it's important to know that it's a red flag for it, not proof).

Your third metric may be problematic in that to get the value you want out of it, you're going to need to commit a lot of time. While 100 changes to a code base could cause a lot of errors and volatility, it could also be 100 steps toward improving the code. What would make me nervous is a lot of commits with comments like "fixed an omission from last build". However, that means reading through all of the commits to see what changes and if it looks good.

So, enough negative, what can you do to effectively judge code volatility and risky releases? It comes down to regression testing. The two metrics I'd use together to judge the risk of a release are:

1) how many changes were made to parts of the code that are not adequately covered by tests.

2) how frequently do your regression tests fail during development.

For the first metric, each change to code not under test is a coin flip. You have no idea what will happen with it. On the second, if developers commonly check in code that fails regression tests and have to fix it after, they aren't focusing on quality in their development practices.

Of course, this means a lot of testing and if you don't have a good suite of regression tests, you'll need to build one up. Definitely start with critical functions and areas that have caused problems in the past. This sounds like a lot of work, but trying to gauge risk off of metrics that sometimes correlate to volatility problems is gambling at best.

Answer 2

My workplace currently does something similar, and assigns a Red/Orange/Green rating to each software release. It's based solely on the number and severity of outstanding bugs - since we have a large user base, we get quite a few bug reports. The formula is something like this -

1+ high severity defects = Red (high risk release)

3+ medium severity defects = Orange (medium risk release)

Otherwise it is rated Green (low risk release)

We use JIRA to track defects, so it's a pretty easy job to run the numbers. If you ARE using JIRA, you may want to check out this fairly cool Risk Management plugin.