How (and in what units) to measure the product scope?

Question

Short version:

How and in which units is it better to measure the product scope (volume of incoming requirements / amount of business functionality) if different parts of product (different requirements / different parts of business functionality) have a different nature?

Long version:

In our company, we want to develop and implement a methodology for measuring the scope of the product.

We want to reach the following goals:

1. To make business requirements measurable and thereby more transparent. I.e. we want to measure all incoming requests for product improvement in some units. These units should not be related with real man-days because we don’t know who exactly will implement these requirements. Furthermore, these units should not be related with ideal man-days because ideal man-days are for the work scope, not for the product scope. For example, we can implement the same requirement in three different ways: we may implement it fast with bad quality of code, slow with good quality of code and extremely slow with parallel refactoring. It’s depends on us, how we will do this item, but we want be honest with our internal customers (another departments of our company) and show the same «price» for all equal requirements (regardless of how we will do it).
2. Calculate productivity of the team. As noted above, the work scope is not equal to the product scope. Productivity in our understanding is the amount of the product scope units that we did within a fixed amount of time (that’s iteration). I.e. the team may do a lot of useful work (to reduce technical debt, for example), but if they don’t implement enough requirements, productivity will be low.

To achieve these goals, we need to choose a unit for the product scope and learn how to measure it.

Story points are good candidates for these kind of units, but we have a problem with them: we have a lot of different technical systems within our product. So, there are many different kinds of requirements: make additional automatization for engineers, create new reports to managers, implement new functionality to end users, etc. Furthermore, one requirement may be related with several technical systems. Managers can’t compare such incompatible requirements just as developers can’t compare tasks for different technical systems (to implement such incompatible requirements). It’s like comparing red and hot. I tried to solve a similar problem here: How to use Story Points, if User Stories are completely different?.

At the current moment, we do the following: we have created a table for the different types of requirements, defined the reference stories (the reference requirements) to each of these types, and estimate other requirements based on these reference stories. There are two (three) big disadvantages in this solution:

1. We have chosen reference stories such that, their implementation took roughly the same amount of time. So, this is a fundamental problem: we have a relationship between reference stories and man-days at the moment when we defined them (we wanted to avoid this situation, see above).
2. As far as we will make changes within technical systems, story points of a different kind will drift apart. In some systems technical debt will increase and related story points will be more «expensive». In others systems technical debt will be decreased and related story points will be «cheaper». As a result, after some time one story point related with different requirements will have a different cost. In that case we will not achieve our first goal.
3. If we periodically normalize story points, by choosing the new basic story of each type, that will have the same man-days estimates, then we will not be able to see any change of the team's performance over time. So we will not achieve our second goal.

Also, I heard about a function point but I didn’t look in this direction. It seems a little complicated. Should I look at it more deeply?

Maybe there is another way to solve my problems?

---

Update after receivied answers (19.12.16)

I meant a volume of business requirements, not a value of it.

I will try to give an example:

Imagine, we are refurbishing an apartment. We have two requirements:

1. To hang a picture;
2. To install a chin-up bar for pull-ups.

Volume of requirements (i.e. product scope):

1. To hang a picture: 1 point, because it is necessary to drill one hole.
2. To install a chin-up bar for pull-ups: 4 points, bacause it is necessary to drill 4 holes.

Volume of work to implement requirements (i.e. work scope):

1. To hang a picture: 1 point, because to make a hole in dry wall is very easy.
2. To install a chin-up bar for pull-ups: 16 points, because a chin-up bar for pull-ups should be installed with wide bolts on load–bearing wall which is usually is made of concrete and I have a low power drill. Accordingly, I know that to make a hole with my low power drill on a load–bearing wall is roughly four times harder than making a hole in dry wall.

Ideally, the volume of requirements should be close to the volume of work to implement the requirements. For example, if I were to buy a good drill, it will be no difference, which material the wall is made of.

The question is, what units and how to measure the volume (not the value) functional requirements, regardless of exactly how we will implement them and how much time we need for this.

Answer 1

You are absolutely on the right track with the Story Point concept. What you are looking for is Business Value vs Level of Effort (Cost).

On a classic user story card we are used to seeing a number (usually in the bottom right corner), which represents the Level of Effort (or Cost) to build the feature. What we are missing is that there is supposed to be another number on the card. A number representing the Business Value of the Feature.

We miss this distinction a lot in software these days. Connecting business value to feature work has become something of a lost art. You ask product people "what is this worth?" and they wave their hands and push back on being able to quantify (I know I was one and used to do this). Or you get the product manager who has gone 20 levels into the industry analyst reports and created this highly detailed forecast which in the end will turn out to be pure fiction (I know, I did this as well).

What you really need is just a dirt simple way of asking "Is this feature more valuable than that feature?" There are two ways I've found to do this with effectiveness.

Team Estimation Game: The first is to just use the same estimation game I recommend for Level of Effort (Cost) estimation. The Team Estimation Game was created by Steve Bockman (there are a couple of parallel developed exercises out there that are very similar). I go into details on why I like this game in this blog and how to play the game in this blog. Steve also published an ebook on Amazon for $0.99.

With Team Estimation you are just assigning an arbitrary value based on the team's (in this case the business side) gut on what is more valuable than the next. "Are new carpets going to add more or less to the home's value than new kitchen appliances?" At the end you have a set of Fibonacci numbers for Business Value (Value). You can then run the same game to get the Cost and you have know now how valuable each piece of work will be.

Buy a Feature This is an Agile Collaboration Framework created by http://conteneo.co/ (Formerly Innovation Games). Buy a Feature is useful when you have a lot of differing views, diverse customers or really want some hard data to back up your decisions. Buy a Feature let's you assign (arbitrary) values to features and then let the customers (internal or external) go through an exercise where they use fake money to purchase the features they want. No one has enough money to buy more than a couple of features, so cooperation must happen to get to a final result. At the end you have a list of purchased features reached by joint decision. You also have something reflecting what the customer really wants. You may still want to assign some Value figure to the purchases features so you can delineate between one that was quickly bought and a feature only one or two people really championed for.

When you are done with assigning Business Value and Cost it becomes a simple exercise to obtain priority. Divide Business Value by Cost and this will give you your priority on the backlog (of course some human tweaking will almost always be needed).

Answer 2

The Agile Approach

Story points are meant to convey level of effort or complexity for a unit of work, and the Connextra format typically used for user stories provides context and scope. However, while prioritization within the Product Backlog is often the main expression of business value, the actual value is more accurately tracked through a separate metric.

Technically, business value is determined by stakeholders, while prioritization is determined by the Product Owner. This is an important distinction, but the distinction is often blurred when people talk about agile approaches because most of the extant literature focuses on the Development Team rather than the Product Owner or stakeholder processes.

The general agile approach is to enable stakeholders to estimate business value based on agreed-upon criteria. This creates a consensus around the relative business value of your high-level product increments, which can then be used to prioritize the work.

Tools and Techniques

There are a wide variety of tools and techniques for implementing the agile approach outlined above. Among them, Mountaingoat Software provides a number of tools for deriving stakeholder-driven metrics of business value. Provided techniques include:

• Relative Weighting

  Relative weighting is a prioritization approach that considers both the benefits of a feature and the cost of that feature. The technique is best applied for setting approximately quarterly goals rather than each sprint.

• Theme Screening

  Theme screening is a useful and easy prioritization technique that can be used to prioritize themes and/or epics against one another. It can also be used to prioritize entire projects or products.

• Theme Scoring

  Theme scoring is a prioritization technique that can be used to prioritize themes (groups of user stories) and epics (large user stories) against one another. It can also be used to prioritize projects or products against one another.

In general, all these approaches provide a numerical result that can be used to measure relative business value. This approach is highly flexible and very effective at building consensus among stakeholders, and only requires that you create buy-in among the stakeholders in the design of the filtering criteria and the choice of methodology. As long as everyone has agreed on how business value will be estimated, there is likely to be wide acceptance of the results.

Answer 3

A different point to discuss must be the risk level of incoming requirements. "Technological readiness levels" can be utilized to quantify the burden and effort probability of the function that is to be added.