2

We are shifting our startup to a data-driven culture, and prioritising what to start testing [UPDATE: i.e. what goes in the product plan]. We have a reasonable number of users, and a working product, and a large backlog of bugs and ideas :)

However, we're struggling with one thing: how should we decide which changes we should just do, rather than including them in A/B tests.

Examples:

  1. It seems fairly clear that bug fixes should just be deployed, and don't need to be A/B tested. Correct?
  2. What about documentation improvements? Should we be A/B testing those, or should we deploy them?
  3. What about UX changes that we have strong qualitative evidence (support requests, user testing) are barriers for users, and that just take an hour or two to fix?
  4. What about UX changes / new features that a paying customer has asked for, but that we don't have any other qual evidence for?

We're having a debate internally about whether it's OK to "just do" some stuff, especially where it's "obviously an improvement" like a bug fix, or whether the time taken to do it is small.

Or whether that's dangerous, and we need to test everything.

The trade-off is that running multiple A/B tests at once gets complex, and clearly sometimes the opportunity cost of adding a new test, waiting for the results, and doing the work to understand the results (which means we can't do other things) is more than the potential benefit/downside of shipping the change.

I've read "The Lean Startup", and "Lean Analytics", but both are pretty vague on this. Both say you should be using design intuition and qualitiative testing at times, but don't give advice on when you should rely on those vs quant testing.

How do other startups handle this?

  • I see this question as being about what goes into the project plan. Apologies if it's the wrong Stack Exchange though, suggestions for where to post it instead gratefully received :) – Richard Aug 2 '18 at 11:05
1

Since you are a data driven start-up, I think the criteria to decide wether you should do A/B testing should be data driven.

Avoid at all costs to rely solely on the time it takes to develop a change as even the smallest changes can have a huge impact on user experience and ultimately in the engagement with your product (I'm mentioning this as your post suggest that you want to use the time it takes to develop as part of the criteria).

Some ideas to decide if a task require A/B testing are:

  • What percentage of the users are affected by the new feature?
  • If you don't have data on a "suggested feature", is there a way to relate it to any other data you are collecting?
  • What's the historical impact on changing something for this part of the product?
  • What's the historical engagement changes expected when changing this part of the product?

Finally, I want to mention that since it's also important to have a good criteria on "things that you just decide to do" it's imperative to monitor and keep a log of what was the reaction to those changes to use it as a feature indicator for future decisions.

  • Thanks! This seems like a good rule for deciding how to prioritise A/B tests, and not relying on a time is a good point. But... I still am unsure how to decide what not to A/B test. What about a bug fix that affects all users, but is clearly a bug fix of something that's broken? – Richard Aug 6 '18 at 14:20
1

It seems fairly clear that bug fixes should just be deployed, and don't need to be A/B tested. Correct?

Incorrect.

At the company I work, a lot of bug fixes are A/B tested. You never know whether the fix introduced a problem that's worse. Maybe your bug fix just makes every request load a little bit slower. You'd never notice that when testing it from the office, and your automated tests are very likely to not notice this either. But if you have millions of customers, and they make many requests before buying anything, the slow down adds up. And if it causes 0.1% of your customers to not make a sale, that's a lot of money you miss. Also, A/B testing can give surprising results. Sometimes customers "prefer" the bug ("prefer" in, "averaged over all users, they buy more").

Obviously, there are many bug fixes that need to be deployed anyway, or for which it's too much of a hassle to make an A/B test. But it's not a given that bug fixes don't need A/B tests. You could even consider "Always A/B test bug fixes, unless there's a good reason not to".

What about UX changes that we have strong qualitative evidence (support requests, user testing) are barriers for users, and that just take an hour or two to fix?

Any customer facing UX change needs A/B testing. If you don't, you shouldn't be calling yourself a data-driven company. You will be surprised what A/B testing reveals. (You've done user testing, but have you done it for your customers in China? In South America? In the weekend? Evenings? The first Monday of a month?)

Here's an example: I work for a company where customers can book hotel reservation. We show whether a hotel offers breakfast, and whether breakfast is included in the price or not. In the UK, some hotels offer a full English breakfast, while others offer only a continental breakfast. There were user request to show what kind of breakfast is being offered. So, I implemented a feature which allowed front-end to show customers what kind of breakfast is being offered. Front-end decided to A/B test this feature. The results were surprising: customers from outside the UK booked more often -- customers from the UK booked less.

Besides, your feature may have introduced a bug, introduced more load, used more resources, etc. A/B testing where initially a small percentage of your customers get the new feature gives you a change to find problems before bringing down your site.

What about UX changes / new features that a paying customer has asked for, but that we don't have any other qual evidence for?

Basically, anything I said above applies to this point as well.

  • Thanks, interesting perspective. "There are many bug fixes that need to be deployed anyway, or for which it's too much of a hassle to make an A/B test... You could consider 'Always A/B test bug fixes, unless there's a good reason not to'". Any views on a good rule for what's a "good reason" and what isn't? – Richard Aug 6 '18 at 14:09
0

It's true there is a very real risk that you can get too caught up in experimentation and move away from "doing work".

Think back to your school science experiments. Measure, change one thing, and then measure again. In other words you should adjust one variable and record the impact it has against your goal.

In other words (a huge generalisation here) you should test one thing at once.

So how do you decide what to test?

The recommendation in Sprint is that you should try to hone in on the biggest risk to your future vision. Small experiments may help drive small improvements but in a start up environment you have a much bigger risk - will you still be in business in 6 months!?

The Lean Startup puts this quite nicely as trying to fit in enough course pivots before you run out of runway. In other words, any start up has a period of time (be it financially driven or motivation driven) before it runs out of runway. What you need to do is answer the biggest questions as soon as possible so that you're as high in the air as soon as possible.

So, what are the biggest assumptions you are basing your business on and how do you change them from assumptions to known facts? That's the experiment you want to run!

TLDR

You should run one experiment at a time, each to test the biggest current risk or assumption to your vision!

  • Thanks. Yes, that's what the books say. Does that mean that we should freeze all other development though - bugfixes, docs changes...? If not, what should we carry on doing? – Richard Aug 2 '18 at 16:53
  • To be clear - thanks for the answer, but this isn't the question I was asking. I need to know what BAU we should continue without testing - not how to prioritise the things that we know we need to test. – Richard Aug 3 '18 at 10:40

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.