We are shifting our startup to a data-driven culture, and prioritising what to start testing [UPDATE: i.e. what goes in the product plan]. We have a reasonable number of users, and a working product, and a large backlog of bugs and ideas :)

However, we're struggling with one thing: how should we decide which changes we should just do, rather than including them in A/B tests.


  1. It seems fairly clear that bug fixes should just be deployed, and don't need to be A/B tested. Correct?
  2. What about documentation improvements? Should we be A/B testing those, or should we deploy them?
  3. What about UX changes that we have strong qualitative evidence (support requests, user testing) are barriers for users, and that just take an hour or two to fix?
  4. What about UX changes / new features that a paying customer has asked for, but that we don't have any other qual evidence for?

We're having a debate internally about whether it's OK to "just do" some stuff, especially where it's "obviously an improvement" like a bug fix, or whether the time taken to do it is small.

Or whether that's dangerous, and we need to test everything.

The trade-off is that running multiple A/B tests at once gets complex, and clearly sometimes the opportunity cost of adding a new test, waiting for the results, and doing the work to understand the results (which means we can't do other things) is more than the potential benefit/downside of shipping the change.

I've read "The Lean Startup", and "Lean Analytics", but both are pretty vague on this. Both say you should be using design intuition and qualitiative testing at times, but don't give advice on when you should rely on those vs quant testing.

How do other startups handle this?

  • I see this question as being about what goes into the project plan. Apologies if it's the wrong Stack Exchange though, suggestions for where to post it instead gratefully received :)
    – Richard
    Aug 2, 2018 at 11:05

2 Answers 2


Since you are a data driven start-up, I think the criteria to decide wether you should do A/B testing should be data driven.

Avoid at all costs to rely solely on the time it takes to develop a change as even the smallest changes can have a huge impact on user experience and ultimately in the engagement with your product (I'm mentioning this as your post suggest that you want to use the time it takes to develop as part of the criteria).

Some ideas to decide if a task require A/B testing are:

  • What percentage of the users are affected by the new feature?
  • If you don't have data on a "suggested feature", is there a way to relate it to any other data you are collecting?
  • What's the historical impact on changing something for this part of the product?
  • What's the historical engagement changes expected when changing this part of the product?

Finally, I want to mention that since it's also important to have a good criteria on "things that you just decide to do" it's imperative to monitor and keep a log of what was the reaction to those changes to use it as a feature indicator for future decisions.

  • Thanks! This seems like a good rule for deciding how to prioritise A/B tests, and not relying on a time is a good point. But... I still am unsure how to decide what not to A/B test. What about a bug fix that affects all users, but is clearly a bug fix of something that's broken?
    – Richard
    Aug 6, 2018 at 14:20

It's true there is a very real risk that you can get too caught up in experimentation and move away from "doing work".

Think back to your school science experiments. Measure, change one thing, and then measure again. In other words you should adjust one variable and record the impact it has against your goal.

In other words (a huge generalisation here) you should test one thing at once.

So how do you decide what to test?

The recommendation in Sprint is that you should try to hone in on the biggest risk to your future vision. Small experiments may help drive small improvements but in a start up environment you have a much bigger risk - will you still be in business in 6 months!?

The Lean Startup puts this quite nicely as trying to fit in enough course pivots before you run out of runway. In other words, any start up has a period of time (be it financially driven or motivation driven) before it runs out of runway. What you need to do is answer the biggest questions as soon as possible so that you're as high in the air as soon as possible.

So, what are the biggest assumptions you are basing your business on and how do you change them from assumptions to known facts? That's the experiment you want to run!


You should run one experiment at a time, each to test the biggest current risk or assumption to your vision!

  • Thanks. Yes, that's what the books say. Does that mean that we should freeze all other development though - bugfixes, docs changes...? If not, what should we carry on doing?
    – Richard
    Aug 2, 2018 at 16:53
  • To be clear - thanks for the answer, but this isn't the question I was asking. I need to know what BAU we should continue without testing - not how to prioritise the things that we know we need to test.
    – Richard
    Aug 3, 2018 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.