Things to consider when a deployment goes wrong

Question

I just launched a big web infrastructure project which has resulted in a lot of inconsistency in the users' feedback on their experiences.

Many of the users are not that web-savvy, and, due to technical limitations, the user experience isn't perfect. At first I thought this was the reason, but there now appears to be some bugs that I'm having problems recreating.

I have made some tweaks to the system in two areas that are not essential but pose a risk and could be contributing to the problem. I have been going through all the customer service emails to spot patterns and then pass these findings on for my development team to investigate.

For situations like this which are likely reactive, what things are good to consider to try and get through the tough times?

Answer 1

The first reasonable step is to learn more about the issues the customers are having. In cases like this the general advice is to collect as much data as possible. Add google analytics events to everything you can think of, take the web server log files and try to recreate the steps the users are taking. With this approach you don't have to try to recreate anything, because you'll have the real data.

When you have the data, take the team and create a list with forecasting : if the issues stay in the system, what is the probability that users stop using your service. Pick the first one, solve it, and continue the data collection and look for changes. If the change is positive, then you and your team are on the right track, and keep solving the issues from the list.

This is called "validated learning" from the Lean Startup Movement. It suggest to do it before going live, but there is no such thing as lost cause. Deliver small changes continuously - I'm talking about days -, collect, and check your data.

Answer 2

I also recommend doing AB testing or variant testing on your website and see the different in these two designs or functionality.