In a data-driven startup, what should we *not* A/B test?

Question

We are shifting our startup to a data-driven culture, and prioritising what to start testing [UPDATE: i.e. what goes in the product plan]. We have a reasonable number of users, and a working product, and a large backlog of bugs and ideas :)

However, we're struggling with one thing: how should we decide which changes we should just do, rather than including them in A/B tests.

Examples:

1. It seems fairly clear that bug fixes should just be deployed, and don't need to be A/B tested. Correct?
2. What about documentation improvements? Should we be A/B testing those, or should we deploy them?
3. What about UX changes that we have strong qualitative evidence (support requests, user testing) are barriers for users, and that just take an hour or two to fix?
4. What about UX changes / new features that a paying customer has asked for, but that we don't have any other qual evidence for?

We're having a debate internally about whether it's OK to "just do" some stuff, especially where it's "obviously an improvement" like a bug fix, or whether the time taken to do it is small.

Or whether that's dangerous, and we need to test everything.

The trade-off is that running multiple A/B tests at once gets complex, and clearly sometimes the opportunity cost of adding a new test, waiting for the results, and doing the work to understand the results (which means we can't do other things) is more than the potential benefit/downside of shipping the change.

I've read "The Lean Startup", and "Lean Analytics", but both are pretty vague on this. Both say you should be using design intuition and qualitiative testing at times, but don't give advice on when you should rely on those vs quant testing.

How do other startups handle this?

Answer 1

Since you are a data driven start-up, I think the criteria to decide wether you should do A/B testing should be data driven.

Avoid at all costs to rely solely on the time it takes to develop a change as even the smallest changes can have a huge impact on user experience and ultimately in the engagement with your product (I'm mentioning this as your post suggest that you want to use the time it takes to develop as part of the criteria).

Some ideas to decide if a task require A/B testing are:

• What percentage of the users are affected by the new feature?
• If you don't have data on a "suggested feature", is there a way to relate it to any other data you are collecting?
• What's the historical impact on changing something for this part of the product?
• What's the historical engagement changes expected when changing this part of the product?

Finally, I want to mention that since it's also important to have a good criteria on "things that you just decide to do" it's imperative to monitor and keep a log of what was the reaction to those changes to use it as a feature indicator for future decisions.

Answer 2

It's true there is a very real risk that you can get too caught up in experimentation and move away from "doing work".

Think back to your school science experiments. Measure, change one thing, and then measure again. In other words you should adjust one variable and record the impact it has against your goal.

In other words (a huge generalisation here) you should test one thing at once.

So how do you decide what to test?

The recommendation in Sprint is that you should try to hone in on the biggest risk to your future vision. Small experiments may help drive small improvements but in a start up environment you have a much bigger risk - will you still be in business in 6 months!?

The Lean Startup puts this quite nicely as trying to fit in enough course pivots before you run out of runway. In other words, any start up has a period of time (be it financially driven or motivation driven) before it runs out of runway. What you need to do is answer the biggest questions as soon as possible so that you're as high in the air as soon as possible.

So, what are the biggest assumptions you are basing your business on and how do you change them from assumptions to known facts? That's the experiment you want to run!

TLDR

You should run one experiment at a time, each to test the biggest current risk or assumption to your vision!