It must be a tough question but I still hope to get some insight.
Tough to answer this question directly, but I can think of a few ways of getting at it - I'd probably want to see multiple metrics used in combination to create some kind of "score"
- Aging analysis of classes / methods (when was the last time this method was changed?)
- Bugs per feature area, or if you're looking for more technical analysis bugs per function point / class / method - I'd caution here that some people might be tempted to use this as a developer performance metric, so tread lightly
- Cyclomatic complexity or function point analysis
- Code coverage / unit test density
- Qualitative code coverage / "WTFness" measure - a mix of engineers familiar with the code and those who are not will yield great insight into things like readability, logical flow, etc - more subjective sure, but so is tech debt :)
hope that helps!
Start by not piling on more technical debt
Write clean code everyday: Whatever code you write or touch you should plan on refactoring and leaving it in good shape. This refactoring should be built into your story point estimate. No one is asking you for an estimate to write sloppy code. As Jeff Sutherland says, "Technical Debt needs to be stopped in its tracks. The discipline of having clean code every day is essential as is completing all testing within the Sprint."
Plan on completing all testing within the sprint: How else can you say that you have created a shippable increment at the end of each sprint? I agree with Mike Cohn's characterization of over-reliance on manual testing as technical debt. In one of my previous projects, we relied entirely on manual testing for a period of time. We needed a "hardening week" every second sprint to get a stable release out. Even then, we uncovered some defects and performance issues only in production.
Intentional technical debt: When you do take on technical debt, do so deliberately. Create a ticket for the technical debt right away and put it in the backlog. When people max out debt on their credit cards to start a business, they do so with the expectation that the returns from the business will help pay off the exorbitant interest charges and the principal.
Unintentional technical debt - architecture and design: Whenever your dev team grumble about poor architecture or design, ask them what they will do to improve it - and create a ticket.
Unintentional technical debt - code quality: @Jon Odo covered some of this in his response. Also see this excellent response by @Thomas Owens on Programmers SE. Use these techniques to identify problem areas and write tickets.
Invest a little time: When you have some breathing time (after a major release?), do time-boxed research stories of the most promising tickets to assess the level of effort and expected benefits. If the benefits are strong and the effort is worth it, then you can lobby with the Product Owner and stake holders to get it priority.