Testing big systems in agile

Question

In the organization I work for we have a HUGE product. It has about ~10 associated scrum teams. We have two types of releases: 1. The big ones 2. Service packs

For #1, we have system QA. For #2, we have service pack QA.

Both test system, that includes areas of other scrum teams.

I am uncomfortable with this approach because then if someone finds something, either it's at 11th hour and we have to run around or they find something but scrum teams are sometimes not aware.

Have you faced such situation? Did you find a way out or this is the same model that is followed in your organization as well?

Thanks in advance.

Answer 1

We face this situation in our AOL Advertising platform today. Like @MrHinsh suggests, the very first thing to do is move all QA into the scrum teams. We are actually in the process of moving all our QA into development roles where they are doing some development and all our developers towards full stack automation.

You need to stop doing big bang releases. Sure you can turn stuff on for the customer in big bangs, it should all have already been running in production though. Code Rots, the minute you stop coding, it moves out of sync with production and every other person coding. So you need to release as quickly as possible.

Here are the things that are working and what we are doing to move farther down the path of continuous deployment.

• No code check in without automated unit tests: A new test must be written or an existing test updated to check in any code.
• Everyone writes tests: It's all code, feature or automated test, so everyone writes. QA's role needs to move more towards helping make sure requirements are well written and understood so good tests can be written.
• - Smaller, more frequent deployments: If you release 100 features every three months, you have 100 things slamming together all at once. If you deploy 5 features every week, you have only five things breaking. If you deploy a feature every day, there is only one thing that could break at a time.

Important distinction you need to wrap your mind around: Deployed - Code it pushed to production, under flags so only development can see it. This allows you to test the code with production, without the customer seeing it. Whenever a user story is done, it should be Deployed.

Released - This is when you let customers see it. You can keep your Releases to quarterly by keeping things under flags. Or even just have some customers see it while others do not. Code should have already been deployed and tested well before you release.

Answer 2

That 1 & 2 exist indicates to me that while your teams are not agile. You should be producing working software with no further work required to ship at least every 30 days.

You need to have all the testing done before you get to the 11th hour. This generally means that you need to incorporate your testers into the development teams, and invest heavily in automation.

Each team should be contributing to the same working increment of the software that is tested on the same cadence.

If you don't have buy in organisationally to make that change then you need to build testing into your development teams in your own. At some point, if you build in enough quality, your QA teams will become a cost center that provides little value.

You should be releasing your software much more frequently and ditch the whole idea of service packs.

Focus on frequent working and integrated software.

Answer 3

The goal with Scrum is to have releasable code at the end of each sprint. Now that does not necessarily mean that you have to do a release every sprint. Just that the code you have is potentially releasable. It could be that your Product Owner does not want to release every sprint, but they should always have the choice available to them.

It may be worth considering having a staging environment that is effectively production-ready code. All production releases would take place from this staging environment and the release would involve no additional testing or configuration. In other words, your staging environment is production ready code that just happens not to be in production yet.

This staging environment represents the end of a release pipeline and it includes the integrated release code from all of the Scrum teams. Each team tests at the unit level, at the functional level plus they run full integration and regression tests on the main code trunk. This is done in each and every sprint.

It is usually impractical to do this using manual testing methods. A better approach is to have automated integration and regression tests that are maintained by all the teams (and run via continuous integration).

This may seem like a difficult approach, but it offers the substantial benefit of avoiding discovering bugs at the 11th hour (a problem you mention in your question). You could also run a parallel release pipeline for your service packs, using a separate code branch.

I would highly recommend you take a look at the book "Continuous Deliver" by Jez Humble and David Farley for a detailed description of this kind of setup.