metrics for bug fixing / learning legacy system

Question

We are a new team of about a dozen developers in our not-very-IT company tasked with reshoring/onshoring/insourcing (bringing into the company what a vendor has developed for us) a large (10M+ LOC?) legacy system. With pretty much zero documentation the approach we are taking is just to work on bug fixes as a way to learn more about the system.

Our management wants to keep track of our progress so they are counting the number of bug report tickets we've closed, which does not seem to be helping. Rather than addressing the highest risk vulnerabilities, the developers are focusing on finding the simplest bugs to fix while avoiding any in-depth troubleshooting.

What could be a better metric for us? There is an opinion that story points are more about new features and they do not really work well for defects, especially since we cannot really know the complexity of these in advance.

Maybe we need not a new metric but a new approach altogether.

Answer 1

Let's get the bad news out of the way first: you can't measure percentage completion of an unknown quantity, so on some level, they're going to have to let go of that.

So let's start with what they want to know. Right now you're measuring bug closures. If you're trying to shore up the legacy system internally, that might be a great metric. If learning how different features work, picking features in priority order (maybe by frequency of use if you've got those numbers somewhere) and giving story point values to the effort required to gain an understanding of that feature. The goal is to have a metric that accurately reflects what you want to get out of the work.

Now the important part: what to do with that metric. Create some timebox. If you already use sprints or something like that, you can stick with that. At the end of each timebox, management can look at what was done/learned and decide if it makes sense to spend another timebox working on it. Indicators that you should stop should depend on the numbers you use, but you'd probably start to see some big change in the measurement. If you're using bugs, you might see a large drop over an iteration or two as you've gotten through most of them and what's left is substantially harder to find (not that you wouldn't fix these bugs, just that it shows you've gotten to explore most of the application). If you're using pointed features to learn, you'll probably hit a point where everything left is a massive level of effort because it's some obscure function.

This should give management some insight in how the team is progressing with the project. It should also take some of the pressure off of developers because you're looking at trends, not measuring raw numbers, so they will hopefully not have the pressure to close a lot of small stuff.

Answer 2

Writing test procedures for the functional description of the system, and writing scenario tests may be a better approach to investigate. But, this requires knowing (or assuming) what the functions of the software ARE. However, if they assigned 10 people for a job, and if they are buying such a large code (10M LOC is as complex as an Airplane software), this investment should be done.

Then running these test procedures, and overseeing the progress of completion of these test procedures may be helpful for the management to forecast the details.

Random scenarios, or other prioritization orders would help approach the conclusion quicker.

Answer 3

I know the horse seems to have left the barn, but did no one think to have the vendor do documentation as part of the development? Might it be cheaper to pay the vendor to do documentation (by which I mean hand over the existing documention... 10M LOC? For reals?) rather than hire a dozen people to start as scratch? By the way, you just became an IT company. Good luck.