We are shifting our startup to a data-driven culture, and prioritising what to start testing [UPDATE: i.e. what goes in the product plan]. We have a reasonable number of users, and a working product, and a large backlog of bugs and ideas :)
However, we're struggling with one thing: how should we decide which changes we should just do, rather than including them in A/B tests.
- It seems fairly clear that bug fixes should just be deployed, and don't need to be A/B tested. Correct?
- What about documentation improvements? Should we be A/B testing those, or should we deploy them?
- What about UX changes that we have strong qualitative evidence (support requests, user testing) are barriers for users, and that just take an hour or two to fix?
- What about UX changes / new features that a paying customer has asked for, but that we don't have any other qual evidence for?
We're having a debate internally about whether it's OK to "just do" some stuff, especially where it's "obviously an improvement" like a bug fix, or whether the time taken to do it is small.
Or whether that's dangerous, and we need to test everything.
The trade-off is that running multiple A/B tests at once gets complex, and clearly sometimes the opportunity cost of adding a new test, waiting for the results, and doing the work to understand the results (which means we can't do other things) is more than the potential benefit/downside of shipping the change.
I've read "The Lean Startup", and "Lean Analytics", but both are pretty vague on this. Both say you should be using design intuition and qualitiative testing at times, but don't give advice on when you should rely on those vs quant testing.
How do other startups handle this?