1

Background

I'm currently working on a product (external facing) and we are on the verge of launch of the product. I want to gear myself up with facing the worst situations since this is the first external product I'm about to launch

Question

Let us assume that I launched the product and I get an email from the management that something is wrong with the product launched in production. What is the best way to handle situations like this ?

closed as too broad by Todd A. Jacobs, Mark C. Wallace, Mark Phillips Mar 6 '16 at 23:34

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    Can you elaborate on "something is wrong with the product"? – Jeff Lindsey Dec 11 '15 at 21:12
  • You haven't provided any risks, threats, or other scenarios. As a result, this question is overly broad. Please narrow it with sufficient context for a canonical answer. – Todd A. Jacobs Feb 28 '16 at 16:53
2

Risk management

If I were in your shoes, I'd conduct a pre-emptive postmortem. Sit the team down and start with the assumption that you have just received the email from management. Ask the team to come up with the most likely reasons why. Then work forward to do traditional risk analysis.

Communications Plan

I would also carefully examine communications policies for web site breach notification. There is ample evidence that the way a company communicates a breach (both internally and externally) significantly affects how serious the breach is, and how quickly it is resolved. I didn't find a real simple example, but if do some research comparing how (for example) Target handled their breach with how LastPass handled their security incidents, you'll come up with good guidance. In your situation, I'd also carefully consult the postmortem analysis of the Obamacare websites. There should be enough data and lessons learned to fill up a Yottabyte disk drive.

IF the site encounters turbulence on launch:

  • Who is measuring what and when? Response time? Database time? Transaction time?
  • Who coordinates response?
  • Who communicates to external stakeholders? Who issues the press release? When? To what press? You probably want only one person to speak to media.
  • Who communicates internally? You probably only want one person to speak to management.
  • Who is in the incident management cell?
  • How is the incident management cell notified?
  • Where do the incident management people report?
  • How often does the incident management cell report?
  • How does the incident management cell escalate?
  • Who has the authority to do what?
  • who can pull the plug and shut down the external product? Under what conditions?

Proper Planning Prevents Pathetically Poor Performance.

2

Mark gives a great answer for how to prepare, spot on for planning ahead.

I've seen this kind of great planning fall down though, by poor execution after launch. When launching a new product I like to use the classic Tiger Team concept.

A Tiger Team is formed of the people Mark calls out in his post. You want representatives from the people who can fix the product, test the product, support the product and make decisions about the product. A good facilitator is also key. A project manager who doesn't even have knowledge of the product can serve in this role.

The Tiger Team meets at the very minimum on a weekly basis. This can go up to as often as hourly depending on how critical response time is (on one company wide launch we had a 24 hour War Room set up for the first three days). The key here is right out of the agile playbook of "inspect and adapt often." You can't just have everyone identified and a communication plan in place. You need to get the key players meeting, talking, and making decisions in as close to real time as possible.

1

It depends on how serious the issue is, what's the management like and other factors as well. Anyways I recommend that you do a soft launch before the real one. To be able to get a sense of what might go wrong and what would the management feedback be like.

So if the product is a website or a mobile app get a group of like a 100 person to use it before declaring to the whole world it exists. And I don't mean tester. I mean a group of your intended target audience "normal users". Then give the soft launch a week or something, I think after that you'd have a better idea on what may go wrong and prepare a mitigation plan to avoid it.

1

You'll first want to gather some information to provide your development team or qa team depending on how your company is functionally structured.

  1. What is the expected vs actual result?
  2. What are the reproduction steps?
  3. Details about their environment? (browser version, ios, android?)
  4. Who are the impacted clients or parties (lets you know approximate severity and who to go to for UAT when you have a fix done)

After you have that information

  1. Provide the above information to your QA/Dev team to verify that this is a valid defect and something is not behaving as designed or expected.
  2. Get an estimate on time to fix from the development team and figure out a sensible release timeframe
  3. Coordinate UAT with the user who reported the issue or whomever their support rep is once the fix is out in a test environment
  4. Follow up with the initial reporter once the fix is live in production

My original answer was not specifically in the context of project management. Apologies.

1

I think your action would depend on the nature and severity of product failure. If the product is still fit for use but have identified defects, then the team you have in place for post-product defect resolution simply needs capture the defect, prioritize with all the others, and then go fix. If the product is not fit for use, recall it and begin your analysis and resolution activities.

Hopefully, you have prepared for and budgeted for post-production issues, including a complete recall...which is likely very unlikely and extreme. Don't over think this. This is typical stuff for all kinds of products.

1

Great answers, here's 2 another ideas, to add to the list:

  1. A roll-back plan.

Before the launch, you should have all the members of the team agree on a roll-back plan.

That would include storing a precise snapshot of the existing system somewhere, as well as who is in charge for the roll back, who gets to make the decision and how to contact all relevant parties.

You may also want to decide beforehand on criteria (besides for management disapproval) of when a roll-back would be warranted.

  1. A staggered rollout.

If you can first roll it out to a small percent of the users, that would enable management to decide if they like it without having to worry about a full roll-back.

For web-based roll outs, you may want to put some system in place (cookies or a URL parameter) that would ensure that management will always see the new version.

These concepts only work if you plan them in advance. They are not fire-fighting methods, post-facto.

Not the answer you're looking for? Browse other questions tagged or ask your own question.