How to investigate who don’t like my product and why not like it? [closed]

Question

I manage an online product, which has a lot of customers. However I feel the adoption rate is not high enough and want to investigate who don’t like it and why not. I believe there are 4 types of customers, and their reasons of not liking the product vary. But I don’t know the distribution of the types of the customers.

I want to design a questionnaire and randomly pick 1000 customers to send out the questionnaire to ask which type of customers they are and why they don’t like the product. The problem is some customers might just not respond to the questionnaire. This could skew the results as maybe one particular type of customer would not respond, leaving me to conclude that this type of customer doesn’t have issues.

Do you guys know how to do this type of investigation to get more accurate results?

Answer 1

What you are discussing is a funnel for a customer feedback loop to iterate on your product.

The good news is that lots of lean startup advice exists to help and guide you solve this problem.

For an in-depth view of The Lean Startup and the measurements and metrics you should track to improve your product you can start with the Eric Reis book, Lean Startup but a better and more pragmatic explanation can be found in the Lean Series by O'Reilly.

Pirate Metrics

Dave McClure popularised the idea of Pirate Metrics, named because they folow the acronym AARRR!

• A cquisition
• A ctivation
• R etention
• R evenue
• R eferall

You have to decide which aspect of the customer funnel you feel is being impacted by the sub-optimal product. Are customers rejecting it early? Are they refusing to refer it? Are they unwilling to pay for it?

It's important to understand about AARRR, because only when you understand all the metrics, you will understand where exactly is wrong with your product.

I have linked the Dave McClure SlideShare above. It is where you should immediately spend your time before progressing.

Customer Personas

You have already identified that you think you have four different types of customer and that is great. If you didn't know it already these are customer personas.

Hubspot has a great primer on how to format and use customer personas here and they also have 20 Customer Persona Questions You Should Be Asking.

Creating the Funnel

1. Segment your customer personas into the 4 personalities you have identified
2. Send your survey to an equal number of each persona
3. Record the response rate of each segment to identified your most engaged customer group
4. Plan remedial actions for the customer segments that are not engaged
5. Analyse the survey data responses and separate the answers into bug fixes, performance improvements, feature requests, customer support and customer engagement (For instance making the Amazon webpage more responsive is different from Amazon increasing the number of email touchpoints after I purchase a product (6 email touchpoints)
6. Identify your super-user community
7. Contact them directly inviting them to participate in improving the prduct directly through a focus group, screen sharing etc
8. Ask the super-users what problems they are facing and how much pain they are facing
9. Triage the painkiller features from the vitamin features (See Painkiller versus Vitamin)
10. Sit down with the Development Team and decide which Feature requests can/should be prioritised using MoSCoW (Must have, Should Have, Could have if we have more time, Would have if resources were unlimited
11. Implement the features iteratively and then use the feature release as a new customer engagement launch pad (Email "You asked for, we listened!) asking for users to try the new feature and engage via social media what they think
12. Repeat ad inifitum until product-market fit is achieved

Caveat

Whilst I have massively simmplified the process, there are lots of areas of expertise in this workflow and it requires considerable amounts of research and planning from the Product Manager. You are aiming for a data-driven approach to iteration with a human touch. Good luck :-)

Answer 2

@Venture2099 has already answered better than I can, so I will answer from a different angle.

There was a company (I do not remember who), that did a survey to see what people thought of a potential future product. They got 10% loved it, thought it was the best thing they had ever seen. and 90% hated it. This seemed like a disaster, normally they would not go to market unless they got over 80% liking it.

But someone pointed out that, that was 80% liking (not loving), and that this would translate into sales to 2% of the population (of the correct age etc).

They took a gamble and ended up selling it to all of the 10% of people that loved it.

However whichever way you go, you need good, un-bias, data. And remember that data and statistics can only disprove your hypothesis, it can never prove it.

Answer 3

If I were you, I would break down my investigation in two buckets:

1. Secondary Research:

   • Find product similar to yours, and learn more about them
   • Find publications, reviews and launch notes of these comparable products
   • Use the above analysis the create SAM or Serviceable Available Market
   • Define Personas that exists in this SAM
   • Identify the Persona that your product is serving. If you can't find your existing customer segments in this above list of Personas, then you don't have a product market fit.

2. Primary Research:

   • Run NPS (Net Promoter Score) Survey. This is almost a gold standard for receiving feedback from existing customers. Note that only a handful of your DAU (Daily Active Users) will take the NPS Survey
   • Run more detailed survey with existing customers or users to gain detailed insights on which feature are the most appealing. Always, include a free-form text box in any survey for gaining qualitative feedback from the survey respondents. The responses in these text fields are great material for opening your presentations to executives
   • Do User Interviews with early adopters of your product to gain their insights. Note that this is not to find the next set of features to implement, but more to find what works best for them
   • Stretch Goal: Do Focus Groups with multiple users (>5) with a goal to brain-storm about your product

Surveys are not the only tool to get a pulse on your customers. You have an army of options, and you should use multiple options to get an accurate insight.

Final Note: Pick and choose from the above list. There is a cost of doing any analysis, and you should stop when the cost exceeds the benefits.

Answer 4

The reliability and validity of testing instruments are extremely hard to create. Even those created by true experts in this field have reliability and validity issues. And then you have the challenge of administering the instrument in way to minimize self-selection bias and other types of skewing biases, including confirmation and experimenter's biases.

Although @Venture2099's answer is quite reasonable, you won't find your answer on this site to the degree where you can build the instrument and administer.

Instead, buy it. There are firms that specialize in this, firms that are staffed by masters and PhD level social scientists and statisticians. Likely very expensive but the results could yield a product change that will enhance revenue that will eventually pay for it, i.e., a favorable ROI. And even using these types of firms you will still have threats of receiving inaccurate findings but doing this on your own will most certainly yield results that are not credible.