TL;DR
Avoid proxy metrics like the plague. Figure out what the feature is supposed to accomplish for the product, why it matters, and how you can measure it directly rather than indirectly (if possible). Then slice up the feature into small, independently measurable work items so that you don't have to wait until the end of feature development to determine if the idea was a good one or just a boondoggle.
Proxy Metrics are Problematic
You are making a rather common but very fundamental logic error with your proxy metric. Specifically, when you say:
[T]he objective of the Dev Mode is to help the developer in the team minimize the time it takes to transfer UI to code.
you are making a rather basic assumption that all "units of UI" are basically the same size, and that all resulting code is of equal complexity. Neither is inherently true, so what really happens is that you end up measuring lead time to turn an arbitrary mock-up into a piece of working code in a way that isn't necessarily transferrable across multiple features.
As just one example, if the specification is to move all elements one pixel to the left that likely has a very different level-of-effort than a specification to add a new tab with multiple panes showing various elements. In other words, you're comparing apples to oranges with this metric, and it simply won't yield the results you want.
Measure What You Care About
In general, from a product management standpoint, what you really want to know is "Do people like this feature?" and presumably "Are they willing to pay for this feature?" Payment doesn't even have to be money; it can be payment in time spent learning or using a given feature, or switching from a product that doesn't have that feature to your product. Regardless, the point is to figure out:
- What you expect as an outcome.
- Why that outcome adds value to the product.
- How you can measure that outcome and/or value as directly as possible.
Many people use a variant of the Net Promoter Score (NPS) as a reasonable proxy for product affinity or loyalty, but there are certainly plenty of other ways to measure these sorts of things more directly. Direct metrics are always best. Consider the following rules of thumb for identifying appropriate metrics, but feel free to adapt what and how you measure to suite the desired business outcomes.
Measure Specific Business or Usability Outcomes
If developers are more likely to use your product if you add a Dev Mode feature, you could directly measure:
- How many developers buy/use/download your product?
- How many developers choose to use your product over some other product when given a choice? (NB: You may need to unpack the reasons, as correlation doesn't equal causality.)
Measure Value
"Value" is often subjective, but can usually be reduced to some quantitative or qualitative measurement. Examples include:
- Surveys, A/B testing, or specialized application analytics report shorter average or mean lead times to completion of a product increment.
- Value may also be measured as a perception rather than a concrete measurement. For example, consider the difference between measuring customer satisfaction or product return rates.
- Value might be directly tied to cost measurements. For example, customers might be willing to pay more for a product with the feature, or more willing to purchase it as an add-on.
Directness
Measure the metric that is most closely aligned with your desired outcome rather than using proxies that deliver implied results. For example:
- If the feature is intended to drive sales, then measuring closure rates with and without the feature would be an example of measuring the goal (e.g. more or faster sales) directly rather than indirectly.
- If the feature is intended to drive productivity, measuring the lead time to completion of a fixed unit of work between a control group and a group properly trained on the feature would be a direct measurement.
- Productivity in software is notoriously hard to measure, so subjective surveys about whether or not users and managers find it improves "productivity" (whatever that means to them) after a suitable learning curve is less direct, but possibly more valuable.
- If tool or feature adoption is the goal, then measure tool churn (e.g. how many people switch to your tool, or away to some other tool) based on the presence of the feature.
- If "adoption rate" (by which we'll assume you mean how many people actually use the tool when it's available) you could directly measure how often people open Dev Mode. How long they spend with the feature within a session is largely irrelevant since if it's boosting productivity it may result in less time inside the feature. Then again, maybe not! But the number of times the tool is accessed is a pretty direct indicator of how often a developer reaches for the tool because they find it valuable.
One Size Does Not Fit All
I wish there was an easier way to tell you what you should measure. Unfortunately, product management is often an exercise in validated learning from an empirical process that you test in the marketplace. Just ask any movie studio whose mega-budget film idea did really well in focus groups and then bombed at the box office. It's really only the empirical results that count in the end.
With that in mind, drive iteratively or incrementally towards validated learning to see if you have a good product/market fit. No matter how great the idea, it's worthless if nobody buys it or uses it. On the other hand, if you don't test the idea, you'll never know whether or not it has real (rather than just theoretical) value.
Small, testable units of value are generally easier and cheaper to learn from than big-bang, all-at-once features. If you can slice your feature using INVEST you're much more likely to get the validation without chasing sunk costs. That said, product management is as much an art as a science, so your mileage may certainly vary.
