How to objectively measure reduction of Time To Market for a diverse portfolio of projects

Question

We, as an RnD department of a small-ish product company, want to focus on reducing Time To Market of the projects that are done in-house.

This raises the question – how do we objectively measure our progress in this regard? The projects that we do vary in scope and risk profile. Since we do hardware as well as software development, the variance might be quite significant – a typical hardware project might take from 8 to 36 months (main constraint being the production time of PCBs and electronic component lead times), while the software projects might take from, say, 3 months to a year or so.

Taking that into consideration, how does one even start measuring "how well we are doing" with regard to reducing overall time to market for our projects? Considering the diversity of the work that we do (and for the most part you could say, that no two projects are the same), how does one come up with a metric or a set of metrics, that would measure that?

Answer 1

I would assume these projects are not unique. They may not all be the same, but there should be common components. If you want to improve what you are doing, work on these components. So basically, look at individual steps in your work and see where you can make things better.

You can start by:

• splitting your projects into common themes (e.g. software only projects, hardware only, mix, etc., whatever types you have).
• identify the steps or activities of the processes you use in those themes.
• measure those parts as you have them now.
• see which parts could benefit from some improvements (not all might; keep in mind the Pareto rule that 80% of outcomes comes from 20% of causes). Focus on the big benefits first.
• apply the improvements.
• measure afterwards.

To get started I would recommend a technique called Value Stream Mapping. In very simple terms, every process has:

• steps in which work gets performed, and
• steps of waiting times or handovers between other stages.

You can look at both to try to improve them. Many just look at the work and try to optimize it, automate it, etc., but when they take the time to map out their value streams, they might see that most times are spent in waiting for stuff (waiting for parts to be available, people capacity to be available to take on new work, information to be available, decisions to be made, etc).

See here for details: https://www.techtarget.com/searcherp/definition/value-stream-mapping

Answer 2

A Portfolio Doesn't Have to be Uniform

It doesn't usually make sense to measure a whole portfolio using the same yardstick if the components of the portfolio don't share some similarities that would make a common measurement meaningful. Instead, you should measure the lead time of each project or program within the portfolio separately, and then if desired you can look at the figures for each in some sort of aggregate.

For example, if you have a portfolio of three hardware products and five software products, then:

1. Baseline your current lead time from initiative to market release today for each of the eight items.
2. Modify your processes in some way that's intended to cut your time-to-market by some target amount, e.g. 15-20%.
3. Kick off some new initiatives, and track against your time-to-market objectives. Ideally, the initiatives should be roughly of similar size and complexity or you'll be comparing apples to oranges.
4. Measure the new time-to-market for each initiative.
5. The delta for each is your improvement or reduction in efficiency for that product.

Once you've done that, and assuming it's a repeatable process, then you can create a new baseline for the overall portfolio. For example, let's say that two of your hardware projects delivered 15% and 20% faster, but the third dropped by 10%. Your hardware portfolio now has an average improvement of only 8.33%, and a mean improvement of only 7.49%. That seems terrible!

However, portfolio management isn't just about averaging. It's also about allocation of risk and resources across an array of projects. Let's examine your options.

Balancing the Portfolio

A better way to look at the uneven results of your portfolio is that if you discard the outliers, your delivery speed improved roughly 17.5%, which is pretty good! Now you just have to decide what to do about the outliers.

From a portfolio perspective, you can decide that something doesn't belong in the portfolio because it doesn't fit or it doesn't meet your risk, cost, or performance goals. Or, you could choose to reallocate cost, schedule, resources, or budget to the "problem" item. So, let's look at your options:

1. Allow projects to deliver on different cadences or through different processes.

   If not all projects have to deliver on the same cadence or use the same process, you can keep the problem project around but treat it differently. For example, use the new and improved process on the projects were it works well, and accept that the previous process was "good enough" for the problem projects. That gives you a 15-20% boost on the "improved" projects, keeps your problem projects at their previous levels, and simply reduces your average improvement to roughly 12%, which your stakeholders have decided is "good enough" for the portfolio as a whole, especially since you now have some parts of the portfolio outperforming.

2. Redesign your processes to improve the whole portfolio more evenly.

   If all projects must deliver on the same cadence or using the same process, then your worst project becomes the bottleneck. That means you need to find a different process that will improve all your projects to an average of at least 15% over the original baseline. The new process didn't improve all projects, so you need to try something different to improve the overall portfolio. Back to the drawing board!

3. Rebalance the contents of the portfolio.

   • If the portfolio as a whole needs to meet certain goals, then you have to decide whether you can improve the problem items in your portfolio, or if you should drop certain projects from the portfolio as out-of-tolerance for your overall budget, risk appetite, available resources, or target schedule.

   • Reallocate resources to target a lower but more even rate of improvement.

     You can move some of the resources allocated to projects that are exceeding your 15% improvement target over to the "problem projects" with the notion of trading less improvement in some areas to more improvement in others. Given the same three projects, you might be able to get all of them to an average of 12-15% reduction in time-to-market, but at the cost of having to deliver some of the outstanding performers that were showing a 20% or more improvement at a slightly less aggressive pace.

   • Drop under-performing projects.

     Alternatively, you might look at the portfolio and decide that dropping your under-performing hardware project and reallocating those resources to the other two projects would allow you to go from a 17.5% improvement in hardware to 25%. Rather than chasing sunk costs or accepting a lower average for the portfolio, reallocating resources to the better-performing projects could improve your portfolio's overall revenue or market share beyond any loss you might incur from dropping the under-performing projects. It might also reduce the risk profile of the portfolio since a faster rate of change reduces opportunity costs and makes it easier to adjust to changes in the market.

As you can see, there's more than one way to balance or even improve a portfolio. It's generally a mistake to treat portfolio management as a zero-sum game where you must keep all the elements of the portfolio in place, or try to treat them as interchangeable.

Define Goals for Each Project, and Goals for the Portfolio

There's no single answer to this one, but in general every project should have some expected return on investment. Even if something is purely a cost center, there's usually some positive result that hopefully justifies the costs incurred. Research and development isn't unique, really, except that much like venture funds a notable percentage of R&D projects are expected to fail, provide low returns, and even generate losses. The goal of a balanced, or at least diversified, R&D project portfolio requires that you take a fairly brutal approach to killing off the most at-risk or unprofitable projects as soon as possible to make more resources available for the more projects most likely to succeed.

As a rule of thumb, 80-90% of venture-backed projects fail, and success is usually considered an exit with roughly a 400-500% ROI. I don't have as much detail on R&D figures, but it's likely quite similar. That's why there's so much emphasis on "unicorns," where the rate of return is 14x or more. It makes up for all the failed and break-even investments, but chasing unicorns is a great way to lose money, too.

There's a good agile principle that applies to portfolio management: "fail fast." That doesn't mean to never try anything risky, because risk and reward are often complementary. This is especially true in research, where you have no guarantees that any given line of research will pay off. However, it does mean that you should never wait beyond the last responsible moment to stop chasing sunk costs. It's much like gambling: you have to define a risk tolerance, and your budget should account for acceptable losses instead of just assuming the happy path.

If a project within a portfolio is under-performing in a modest way, if the overall portfolio can absorb the cost of re-balancing then go right ahead and do that. By spreading the risk, costs, and resources across the whole portfolio, you lower your overall level of risk but you also lower your overall potential for returns.

On the other hand, if a portfolio project can't be salvaged, or exceeds your risk appetite or acceptable loss threshold, then the smartest thing you can do is not allow the failure to continue, dragging down the whole portfolio. If you have $10MM to spend across ten projects, they all break even at $1MM in returns. However, if 9/10 projects fail and you've spent the entire $1MM budget on each, then your "successful" project has to provide a 10x ROI for the portfolio to break even.

Some Examples of Cost-Balancing a Portfolio

What if, instead of waiting until the last dollar was spent on each project, you decided to kill off any project that wasn't on track to break even by the time you were $500K in? Even if the same 90% of projects fail, now a single successful project only has to return $5.5MM for the whole portfolio to break even.

Even better, if you monitor and rebalance the portfolio frequently, you might be able to detect inevitable failure sooner for some projects, and redistribute those resources so that fewer projects fail in the first place. For example, let's say that you find four of your original ten projects that are doomed after only spending $250K each on R&D. You now have an extra $3MM in money and other resources to spend on your more promising research candidates, increasing the likelihood that some of them will succeed. Even if half of the remaining portfolio items still fail, you now have three successful R&D outcomes that only have to average a $3.3MM ROI for the portfolio to break even—even less if you killed off the additional three failures at the earliest responsible opportunity!

You started with 10 projects at $1MM each. You saved $750K per project ($3MM) by killing four that were unlikely to succeed. You have more confidence in the remaining projects, so your budget for them is now $1.5MM since you can redistribute up to $500K more to each of the remaining six projects as needed. Despite freed-up cash reserves, your defined risk appetite still requires you to kill another three projects that seem likely to fail at the $750K mark. 7/10 projects have now failed, but you've still saved $2.25MM of your original portfolio budget!

If the remaining three projects succeed at their original $1MM price tag or less, you reduced risk for the whole portfolio and made additional resources available to shore up a few projects that just needed more resources than originally planned. Here's the trade-off: more projects "failing early" increases the available budget for the remaining projects, while fewer failed projects in total means the whole portfolio can be profitable at lower rates of return from each project.

R&D is never guaranteed to be successful, either as a project or resulting in commercially viable product. The best you can do is to to define your research goals and your inflection points for determining whether a given line of research or development remains viable or not. By avoiding the sunk cost fallacy, you improve the chances of a reasonable percentage of your research projects leading to meaningful outcomes with a more achievable ROI target.