How can we estimate the learning curve for new technology in a project?

Question

I work with embedded systems and sometimes there are some technologies that the people involved with the project have to learn before applying to the product.

We are three people that never worked with an RTOS. Our company has acquired a RTOS from Micrium. They provide books and documentation that total almost 3,000 pages to read and learn, and offer three days of hands-on training that we will participate in. Our company's desired deadline is one year to develop this new product, and they think this is possible.

This is the problem. They "think" rather than "know" that it's possible. Maybe they are right, or maybe not. I think this learning curve is difficult to estimate, and in consequence the rest of features of the product are difficult to schedule.

Is there some way to estimate the time it will take to learn this new system?

Answer 1

TL;DR

Firstly, there is no canonical answer to this, but there are some pragmatic solutions. Secondly, you can't estimate what you can't measure. You therefore need to devise a proxy metric to create your estimate or to graph your cone of uncertainty. Thirdly, the project's assumptions and risks (particularly including schedule risks) should be defined and documented.

Bound Infinite Ignorance

By definition, the scale of what you don't know extends to infinity. In other words, you can quantify what you do know, but you can't really quantify what you don't know.

Within any given problem domain, though, the question is "When will I probably know enough to get the job done?" That's more answerable; pragmatically, you estimate your team's level of ignorance on a subject by subtracting what they do know from what they think they need to learn, then estimate the level of effort involved in learning a sufficiency of that subject.

Spikes and Proxy Metrics

Use story spikes or proxy metrics for a first pass at estimating learning effort. An example of a proxy metric might be how long it will take the team to read 3,000 pages of documentation. That's not measuring learning directly, but it's an approximation of a measurable learning-related task.

Story spikes are another useful tool. For example, you might pick a minimalist, throw-away feature to develop for the purposes of estimating learning effort. You might spend four weeks trying to build the feature the same way you might take an open-book test, building, reading, and learning as you go.

Whether the feature is ultimately finished or not doesn't matter; nor does it matter if the feature is inherently useful. The goal is to get a feel for the time and effort that will be needed to master the knowledge domain sufficiently to deliver some set of defined features. Granted that it's only an estimate, it's still based on empirical results and will be more useful than an arbitrary guess.

Worked Examples

Proxy Metrics

Let's say that each team member can usefully absorb 100 pages per day of available documentation, including the time it takes to read it, try out some examples, and try to make sense of it within a cognitive framework. For each person to read the entire body of documentation without skimming would therefore take 300 business days, which is significantly in excess of the roughly 252 business days available each year.

Story Spikes

Instead, let's say that you decide to build a toy project where the team will build a simplistic application that counts the microseconds between button presses. Each team member will focus on an area of specialization within the documentation, and try to read/learn/build the feature as a group within a four-week time box.

At the end of the time box, the team should have a sense of how easy or hard it will be to build the feature set defined for the project while learning as they go. Most importantly for your purposes, you can use this estimate to determine whether a year is a reasonable time frame or not for the team to deliver the desired feature set.

For example, the team could:

1. Do a test project with the new technology.
2. Use the results to estimate the Product Backlog.
3. Use the team's historical velocity to estimate the schedule at an 80% confidence interval.
4. Compare that schedule estimate against management targets to see if the schedule is reasonable and the pace of development seems sustainable over the life of the project.

"Cone of Uncertainty" Adjustments

Whether you use a proxy metric or a story spike, you also need to account for the team's uncertainty by presenting the estimate as a range. In software development, the cone of uncertainty is typically assigned as a "fudge factor" of 25-400% of the first estimate. (Boehm, B. Software Engineering Economics, Prentice-Hall, 1981.)

Management targets are often aggressive, and may ignore the team's scheduling estimates or the cone of uncertainty. That's why:

1. The project manager must make it clear that an estimate is an educated guess and not a guarantee.
2. The project's assumptions are clearly documented.
3. The uncontrolled risk associated with the uncertainly must be documented as an accepted risk that's owned by the management team.
4. The project has controls to detect when the projected schedule is out of tolerance.

The key here is to ensure that the uncertainty and lack of domain knowledge is factored into the project plan. As long as you don't hand-wave it, estimating your team's uncertainty can serve a useful purpose in helping the organization to define a realistic project schedule.

Answer 2

There is no magic formula here that would apply to all projects. What i would do is to get the estimates and them compute, like this:

First, make a plan based on the assumption that people are familiar with RTOS.

Then, get those answers:

What is the training required to learn RTOS? After this training, how much time people will need to be able to be efficient?

Add the required time to your project plan and then add some buffer to your project.

Very important: You need to make sure that all the assumptions are well known by all stakeholders, this way, whan an assumptions fails to be true, you will need a change request to handle the change in the plan.

Example:

Add to your project those assumptions:

• The assumption is that training on RTOS will take XX days and that after XX days the team will be proficient.
• In case any assumption fails to be true the changes to the plan will need to be handled by a change request