In real world, business initiatives always take higher priority as there are associated ROIs and deliver something tangible to the users. But there are technical initiatives and projects that need to be done to keep up with the different versions of software, upgrading to a newer platforms, architecture re-factoring etc.,
I'm going to pick on @jmort253's answer, because I disagree.
ROI initiatives aren't the only kinds of project, nor should they be. Chris Matts (the analyst behind BDD, Feature Injection and Real Options) found that 80% of one of his CIO's projects were concerned with protecting existing customer revenue, rather than trying to increase it - essentially, stopping their customers leaving for a competitor.
A lot of projects also simply provide options for the future. These are often very hard to price, but very, very expensive if they're not done.
Most technical initiatives are done for this reason; to provide the business with options to change in the future. Some of them have more direct business concerns which are more easily phrased, like increasing current system performance to cope with growing demand. Others may be relevant to different parts of the business - changing from Fortran to Python to attract better talent and make it easier to recruit - but in the long run, that's still about having options in the future.
Technical Debt is, and should be, a PM's concern
Chris Matts and Steve Freeman came up with a lovely analogy. "It's not like a credit card. It's like an unhedged call option. It's like you've promised to sell all these chocolate Santas at Christmas, and then suddenly one year the price of chocolate is really high, and you have to sell the Santas anyway because you made that commitment, and now you're bust." As long as nobody makes the call, technical debt doesn't matter.
The problem is that every project has something new about it, or that project wouldn't be happening (see "Waltzing with Bears"). So every project has changes that get made as discoveries happen as a result of feedback from the new thing (see the complex domain in Cynefin). And every project is therefore seriously at risk from being called.
(Every project is also difficult to estimate for this reason; you can't estimate something you've never done before.)
The option to change is usually an unstated a business goal
I worked with one company that had a single class of 10,000 lines that took Visual Studio 5 minutes to load. When the business found out what poor quality code had been produced, they said, "Why would you ever do that? We expected you to push back if we were making you do that!"
This unstated goal is also, usually, the core goal of most projects which are replacing legacy systems. The legacy systems have become too unwieldy, and can't be changed to meet new requirements and architectural demand, so a new system is created. If I had a shiny English pound for every time I've seen a replacement system team abandon the core goal in order to meet some arbitrary deadline, I wouldn't need to work again.
By calling out this unstated goal as an explicit one, everyone involved on the project can talk about it rather than assuming it's happening.
Educate the business about the cost of technical debt
Technical debt only happens in the face of time pressure (or as a result of bad habit or lack of skill, which are a different problem, solved by having time to learn how to do the job well... so, time pressure).
By keeping track of the growing cost of the debt and making the business aware of it, you can help to show them the value of the options they're losing. This could include things like your best developers leaving, and the cost of re-hiring; how much extra time the devs reckon it took to create a new feature as a result of technical debt; how much more effective the team is when they're given the chance to take pride in their work instead of bowing to business pressure.
Also educate them about the alignment trap. "Companies in which IT was highly aligned with business but not effective were considerably worse off than companies in which IT was less aligned and merely kept the systems running."
Pay back technical debt one piece at a time
In this place I agree with @jmort253. It's even better if you can avoid falling into debt in the first place, but usually there are different skill sets on a team, people learning to code well, etc. - so having the ability to refactor as you go and help educate other developers is more important than getting it right up-front.
If the team feel pressured to churn out new features, though, they'll very quickly abandon the technical debt. It can't be seen or measured easily, and as such is usually the first victim of any pressure.
A PM's responsibility in this situation is to push back, educate, and help the business cut scope instead.
As a project manager, the short answer is, you don't. This is really a job for the technical team, which includes functional managers and the actual developers working on the project. If you're doing project management correctly, then you should be getting all of your estimates from the developers themselves.
However, if it becomes a noticeable detriment to productivity, it may become a project management issue.
However, here is where this can get tricky and why sometimes it is helpful for a project manager to have enough experience in the subject matter to know the difference between a good estimate and a bad estimate. New developers, fresh out of college and eager to apply their newfound knowledge, will sometimes be the first to suggest a complete rewrite to the database layer because it's a cluttered mess.
Sure, the job may have been to write a feature that lets the customer service team sort customer data by creation date, something that helps the team improve retention efforts, but this eager dev attempts to take the issue scope to another level. By prioritizing technical debt the wrong way, the actual feature being developed is removed from focus, and management is justified in stopping this in its tracks. If revenue isn't being generated from a development operation, then there's a problem with the balance.
As a developer, I've worked with some code that I've had a pretty strong opinion about. When I told my project manager about it, he said he didn't care. He didn't want to think about it, and that it was my job to include this in the estimates in a way that didn't conflict with business goals. I've been doing that ever since, and I manage my own technical debt while still keeping management happy with the progress. In many cases, it actually makes the development faster because the impediments that would normally slow me down are removed.
Ultimately, the solution to managing technical debt and balancing it with company initiatives is to break it up into manageable, bite-sized chunks. If the database layer needs a rewrite, start with the new feature. Add it in a manner that scales, and then next time a new feature is added fix a little bit more. You should only commit to paying back a small portion of technical debt so that it doesn't stand out as a red flag to management, similar to how you should only pay down enough of your debt to where you can still pay your rent and eat your meals. ;)
In short, the way to balance technical debt against business goals and initiatives is to refactor as you go. The code will never be perfect, and that isn't the goal. Instead, the goal is to manage the technical debt to keep it from becoming an impediment, because when it impedes progress and becomes a threat to the project, then it is a project management issue.
Fascinating discussion, and very relevant to me. I believe that technical debt is a PM's concern.
One could argue that a project is a coordinated effort to produce a specific outcome/change; if the project produces the outcome, the project is successful, and technical debt doesn't enter into the equation. But I think that is shortsighted; at a minimum, the PM is obliged during the closeout procedures to document lessons learned and update the corporate process assets. Technical debt should fall into that category. Beyond that, the PM has an ethical obligation to do the best for the company within the constraints, and leaving a company saddled with unknown, undocumented technical debt is unconscionable.
Alternatively we could consider that technical debit is a quality control problem. Technical debt is a aberration that falls within control limits, but if it can be eliminated, the quality (and presumably value) of the product rises.
Multiple ways to look at the problem, but each of them convinces me that although the PM can ignore technical debt, prudence dictates that the PM estimate, document and manage technical debt.
I think the key challenge is establishing the value of technical debt. I think that companies/industries/teams should establish reference values and refine as they go. Just as the PM is obliged to assess & estimate the impact of every proposed change on schedule/scope/quality, the PM should assess & estimate the impact of technical debt. I think that an interesting corrallary to @Jmort253's question is "Change proposals go to the CCB. Who is accountable for changes that are intrinsically outside the scope of the project?"
As I said, excellent, thought provoking question.
I tend to disagree with the basic assumption that technical initiatives are not business initiatives. They may be different from rolling out a new product or an advertising campaign, but:
Where technical initiatives tend to have problems is that they aren't easily understood by decision makers who tend to come from the marketing/sales/"business" side of the house. Basically they aren't as "sexy" as a new marketing campaign or partnership.
The best approach to getting a better focus on "technical debt" is to do a thorough business case for addressing it. How much will addressing the debt cost in time, money and effort? What is the payback period given the expected labor savings, increased sales, increased customer retention, increased market share, etc once the initiative is implemented? How does this compare to the costs and benefits of maintaining the status quo? Getting this information together will take some time and effort, but if it is done well it should be obvious to you whether or not the technical debt needs to be paid now or if payment can be deferred.
I would start by Identifying the ROI that the technical debt delivers and discussing this with the business. If you can describe both value work and what John Seddon calls 'Failure Demand' in the same terms and using the same metrics then you can agree how to split your capacity across them.
Example - Architectural Refactoring
We have a piece of configuration information which is baked into an app (I know, I know...). This config can go unchanged for weeks and then suddenly need changing.
If the config doesn't get changed then ads stop appearing in an application and we stop making revenue from it.
The quick and dirty solution is to change the config in the code and redeploy. This is repetitive and error prone and ultimately not sustainable.
The sustainable solution would be to factor our the config from the code and provide a mechanism for changing it independently from the code. This will take longer the first time round but will then make future config changes much quicker and less error prone.
Guess which one we have done twice already? That's right the quick and dirty. This is because the ROI on the feature is immediately visible or rather, the Cost of Delay were the work not to be done is very high.
The sell we have to make to the business is, let us spend time refactoring this to address the technical debt. That way, next time we need to change the config it will be quicker than it currently is, so quicker ROI in the future, but we will need to take some capacity away now to address the technical debt. It is deferred ROI.
Example - Architectural Refactoring as complexity increases
Sometimes it is hard to describe the technical debt to the business, to them it is just an implementation detail or something you should not have accrued in the first place.
For example, as something grows in complexity it requires refactoring. In this case I do one of two things.
I either cover the cost of the refactoring in the delivery of the feature which has required us to go back and look at the code in question. I always call this out to the business either to tell them that the feature will take longer to deliver than similar features have in the past, because we have reached a point where we need to refactor.
Again I describe this to them as deferred ROI. There will be a greater upfront cost but by doing the refactoring now, future work in this area will be cheaper.
The other approach I take is to treat the development team as a customer of itself and reserve a percentage of capacity, agreed with the business, in each iteration for reduction of technical debt.
Example - Platform Upgrade
For the example you give of upgrading to a newer platform I would again look at how you can describe this in terms of ROI, or impact on ROI.
If the existing platform is no longer supported by the vendor then the business is exposed to the risk of defects discovered in that version of the platform not getting fixed and so impacting on business functionality which depends on it.
This impact will be an increase in failure demand, that is, customers will be less able to get value out of what you as a business provide as problems with the underlying platform are getting in the way. This can be articulated as lost revenue, or potential lost revenue so the ROI is to upgrade the underlying platform.
Prioritising such an upgrade can again be done by looking at the Cost of Delay. If you were not to upgrade the platform would there be any impact on your existing products and services? Would it prevent you from launching new ones?
If problems in the existing platform are generating failure demand now then the Cost of Delay is high and you have a compelling ROI based argument for doing the work sooner rather than later.
If you are currently unaffected by problems in the existing platform but need to upgrade in order to add new functionality then the ROI of the new functionality will determine the ROI of upgrading the platform.
I know there's an answer for this and I mostly agree with @Lunivore and @jmort253 answers but I thought I would chime in on how I have seen and heard technical debt handled?
In our company we recently started trying to allocate dedicated week-long sprints that we focus on technical debt. Unfortunately this hasn't worked super well when there are certain features that go over the course of multiple sprints, in which case it's impossible to prioritize low-priority fixes vs features that need to ship.
Kind of building on above, I believe it's important to show and communicate communicate how technical debt is affecting the team's velocity. This naturally starts with being able to know what your team's velocity is, and then showing it is simply a matter of process.
Recently we have started costing out all issues with story points (whether it be features and the small tasks associated with features, or just nitpicky bug fixes) everything is costed out with a certain value. After a period of time, our manager is now able to give a ballpark range of how much bandwidth each engineer has and how much time they can allocate to fixing certain bugs. When we first started this, it was naturally really bumpy because we would forget that there was more to a task so its generally better to overestimate than underestimate.
After some time though, we were able to give a good sense of what we can fit into the sprint and what we can't and we have been given dedicated time to fix bugs that makes sense (hey you have this feature but from what was costed out you seem to have time for some small bugs?). I think we are still working on a good rhythm and we have juggled between alternating people to fix bugs or just dedicating time for the entire team (especially useful if you have new devs) -- maybe I'll chime in again when we have narrowed that down but we have definitely seen an improvement in how fast we do things when given the time to fix the things that no one sees and it's nice that everyone can "see" it too now.
In regards to responsibility, I don't necessarily think it's the PM's responsibility? Our engineering manager and tech lead drives managing technical debt while our PM drives the projects and timeline of features. That being said, it's important there is dedicated time for the PM and the manager to coordinate and set expectations.
