Best strategies for dealing with integration testing bottlenecks?

Question

We were recently having a debate around what is the ideal and most effective / efficient way to do integration testing in complex environment with large number of applications to try to get software delivered.

In an organization that has a number of different application teams doing work and releasing independently. These teams independently are "agilish" and they do unit testing and functional testing early on in their cycle and can deliver when their changes are in their own application silo without any other impacts. That being said, when it comes to integrating more complex changes that have larger impact on other apps and teams, and having to get into the environment with every other team that is making changes, it causes large perceived bottlenecks in our ability to do proper integration testing.

The problem exists where there is a perceived "integration testing bottleneck" due to a few reasons:

1. The perception that a separate set of folks (outside any of the individual teams) need to do this work
2. Its quite possible that each team is doing "too much dev" as they don't realize that the system as a whole can't absorb so much change given the integration testing that gets required so we are wasting time developing features that we can get shipped.

The only recommendation so far is to have teams push their changes into the integration environment as soon as possible (so changes are visible to all teams as early as possible and issues get detected as early as possible) but I wanted to see if there are other recommendations for how to "optimize" in this situation around large integration testing.

Any experimental and unorthodox ideas are welcome (technical, organizational, etc). Also, any case studies or references to companies who allegedly do this well and have best practices would also be great.

Answer 1

TL;DR instead of dealing with the bottleneck, focus on its root cause: teams not having integration testing as parts of their "definitions of done".

These teams independently are "agilish" and they do unit testing and functional testing early on in their cycle

First, they should be agile, not "agilish". It will provide lots of value alone, before getting to the integration part. The rest of the answer assumes that you have more or less working agile environment.

These teams [...] can deliver when their changes are in their own application silo without any other impacts.

This implies that they have a stable definition of done and can get stuff to that level. Now, integration part comes in. Going by the lines of the answer to a question with similarities (https://pm.stackexchange.com/a/12792/13461), a combined definition of done for the teams involved should cover integration as well. I.e. whatever changes a single team does should leave the whole system functional and should not break anything.

The only recommendation so far is to have teams push their changes into the integration environment as soon as possible.

You don't have any other option to test integration other than actually testing the integration. Delaying that creates an inventory of untested code that is bad.

if there are other recommendations for how to "optimize" in this situation around large integration testing

It doesn't really matter if integration is large or small. If you have changes coming that affect multiple systems, you should be delivering working increments often, and therefore test integration often. This way you will not be accumulating untested parts, and there won't be a bottleneck in the first place.

best practices

The stuff above is not a result of a thought experiment and comes directly from the Scrum Guide (http://www.scrumguides.org/scrum-guide.html) (if you are not using Scrum, it is still a good point):

The heart of Scrum is a Sprint, a time-box of one month or less during which a “Done”, useable, and potentially releasable product Increment is created.

If the thing that team delivers did not undergo integration testing, it is not done, not useable an cannot be released. Therefore, such sprint did not finish successfully.