Predicting software errors

Question

There are two distinct projects where we are trying to define a bug prediction model.

We used bug at file level and defined co-factors:

• Number of changes per file,
• Developers who left per file.

Project 1 presented for both positive co-factors.

Project 2 presented changes per file positive and developers leavers negative.

I can interpret in project 1, more changes in file more bugs. More developers leave (lose know-how) then more bugs.

How come project 2 we can have the less developers leaves the project, the more bugs?

The only thing come in the top of my mind is that people are not skilled, or that the project is too old and people must be very careful to touch the project and to not break anything.

Answer 1

Don't Perform Bad Science

The fields of computer science and project management currently have no models that can accurately predict software errors. There are various theories and models, but none of them will provide you with guaranteed results. See cyclomatic complexity as just one example where there is disagreement about the measure's correlation to defects.

In your case, you are attempting to perform statistical analysis without sufficient thought given to either test construction, controlled variables, or the null hypothesis. You are also confusing causation and correlation. Even if you have a validly-constructed test with adequate controls, and even if you are able to reject the null hypothesis, you would be extremely hard-pressed to use either of these projects as direct proof of cause and effect.

Introspect Your Projects

A better use of your time would be to look at Project 2 carefully to see why you have such a high defect rate. Inspect your processes, form a hypothesis, and then attempt to adapt your processes to reduce the defects.

A good project manager won't spend time attempting to form a "universal theory of everything" based on extrapolations from two projects. Instead, an experienced project manager would spend that time fixing Project 2 to improve its chances of success, or communicating the likelihood of failure to the portfolio manager so that a strategic decision can be made about whether or not the project should be terminated as a cost- or quality-control measure.

Answer 2

It may or may not be valid to assume the developers on Project 2 are not skilled. You would need to gauge their performance across multiple projects before you could come to a reasonable conclusion as to their skill level.

Perhaps with regards to Project 2, it does not matter who is working on it. Perhaps the project is so complex that, as you say, anyone touching it is likely to break something. I would focus more on that issue than anything else.

Answer 3

Wrong approach!

You can't predict software errors, the same way you can't predict car accidents.

As programmers gain experience they learn from their previous mistakes, but make more sophisticated errors.

Number of changes per file

Not a good indicator of anything. The changes could be from additions or bugs, typos or logic errors.

What is a file? Some files have multiple functions and lots of lines of code. others may have a single function, with even more lines of code, or maybe almost no code. Some others may be header files that don't do anything, but are needed for compiling.

(If the above means nothing to you, then you need to get up to learn about it, as I wrote here.)

Imagine deciding the value of a car based on how often it was washed.

Developers who left per file

I have no idea what you are trying to infer from this metric. Sounds like trying to predict the popularity of a fast-food joint based on how many quarters they gave as change.

The rules about bugs is to test from early stages of development, and to keep a 1:1 or 2:1 ratio of programmers to testers. Then you can safely assume the testing-debugging stage will take as long as the time originally estimated to write the code.

The later you start testing and the fewer testers you have, then more bugs will live and start growing in the software and the debugging stage will take longer. Then you can double or triple the testing/debugging time.

Why?

• Because fixing code that was written recently (today) is easier than fixing code written a while ago (days, weeks or months). Code that's not fresh in a programmer's head needs to be re-read and understood.

• Newer code is often based on older code, so if there's a mistake earlier on, it may waterfall all the way down and cause trouble everywhere.

Lastly: Number of bugs is not a good indication of anything, unless you factor in their severity, importance and impact. Both a typo and intermittently crashing the system are important bugs. One will take a few minutes to fix, the other may take a real long time, as intermittent crashes are almost impossible to recreate and find.