How is incident management done in a large scale Scrum environment?

Question

How do large organizations, such as Amazon, Netflix or Spotify organize 24/7 support. These companies are known to have agile development models, which include "You build it, you run it". That means, there is no operations department which takes care of the application / feature once it is available to the end user. The team itself does maintenance and support. I know about all the advantages of this method, but I see some problems in terms of execution:

• Agile teams are designed to be quite small, so there is just a limited number of people who can deal with an incident. These people are usually not available 24/7 as they are on vacation, sick or simply not reachable.
• Theoretically, every developer needs access to production, meaning deployment, databases, filesystems etc. Are all these permissions just granted and it is only about trust?
• Who must be involved to do a fix, rollback etc.?

How is incident management done in a large scale Scrum environment?

Answer 1

According to Netflix's blog they do, in fact, call up their teams at early hours of the morning if their code breaks.

Now, that is not to say that all problems have to go to the coding teams. Some notable examples are:

• Problems solvable through normal application operations: I call up because it says my account is expired and it's not. This may be caused by a bug in the software, but a front-line support rep may be able to resolve it for me by manually reactivating my account and then the team looks into the bug when they come back in to work.

• Infrastructure with them (and many other large companies) is out sourced. If I recall correctly, Netflix uses AWS. Network or hardware problems are handled by their vendor. If it was in-house, they'd probably still have an operations team that at least handled hardware maintenance. It's worth noting though that many of these applications are designed to be incredibly resilient. A server going down (or even a handful of servers) is unlikely to have any visible impact on the end user. This is another area that Netflix has done amazing work with their "Simian Army".

Team Code Ownership It's also important to understand that silo'ing is your enemy here. In most of these organizations, a team owns a problem and you can get anyone on that team to help you with a critical issue. That helps with things like vacation or something just sleeping through their phone's ringer. If you haven't seen them, take a look at Spotify's blog videos on how they structure their teams to support each other.

Answer 2

Many organisations have a rotating on-call phone for someone allotted to "manage the issue to completion". You only need one person to have the phone, and if your normal customer support need to escalate then they call that number.

The individual on the end (likely a member of your development team) is responsible only for managing the issue. So they triage and wake up who they need on the teams to get the issue resolved.

Hopefully all your infrastructure if already in the cloud and you have automated release as well as production analytics and exception logging. Issues should be able to be resolved without server access. However you should create culture where you can trust them with server access.