Organizing testing in a Scrum(ish) development project

Question

First: I'm trying this forum, even though this is more a test management than project management question. The test/QA exchange seems to be focused on technical, hands-on test questions and issues.

The core question, copied from the bottom by request: I guess what I'm asking is suggestions on what/how to test what/where, and how to organize the testing. With regards to customer/user involvement, but also in the development team itself (with the developers and testers). I will try to clarify the question further if necessary, of course.

I'm wondering if anyone has any ideas and/or thoughts about a development and testing setting which I'm struggling a bit with, test- and test management-wise.

I come from a more traditional setting, where we had separated System Tests and Acceptance Tests. The System Testing was done by the testers/QA engineers, while the Acceptance Testing was done by the customer and their power users/testers. Usually the Acceptance Test had a dedicated test manager, and the System Test Manager handed their test report over to him/her and the product owners.

I've also been a tester and test manager in Scrum(ish) teams. But this wasn't too different from how we used to do it in more waterfall-based settings. The Test Manager and/or testers created Test Cases at the beginning of, and during, each sprint. These were created from the requirements, and run during the sprints as soon as the development tasks were finished and checked in.

During the sprints, especially near the end, we ran the regression test suite. This was a collection of existing tests as well as those new tests which we decided to include in the suite. Pretty standard.

The sprints also often ended in a demo for the customer/product owners, although this wasn't always possible due to possibly poor sprint and functionality planning.

At the end of X sprints we had a System Test phase, which included a large collection of tests made during the sprints, regression tests etc.

Then the Test Report was created and "everything" handed over to the test and project managers for the customer.

Now, I find myself in a development setting/team where things are a bit different, and I'm struggling a bit to find out what, how, when etc. Both when it comes to the actual testing, as well as the metrics, management etc. It's the development of a new version of a system already in production, with both new features and bug fixing.

• The development process is sort of scrum(ish), in that it's organized in sprints of two weeks (which I find a bit on the short side). No daily stand-ups, though - just a weekly meeting.
• We're using continuous deploy and integration (to the test environments). (Release processes to prod. env. are not yet hammered in stone.)
• The project uses TFS and the specified work flows and types of work items default to that system: EPICS, FEATURES and USER STORIES (as well as bugs and tasks, of course).
• There is not yet a good focus on Unit Testing among the developers. This is a work in progress.
• There is no whole-team planning phase at the start of each sprint; The development/team manager decides which User Stories to take into each sprint.

This is how the project (at the moment) is structured:

EPICS: I haven't figured these out fully yet, and neither has the development leads, or so it seems. A work in progress.

FEATURES: This is a collection of user stories (usually no more than a handful.

USER STORIES: These are very small and specific. Example:

• US1: As an X-user I am able to view a list of each tenant for a customer.
• US2: As an X-user I am able to edit a tenant entry for ...

The QA-team/testers are, as of now, testing the user stories. At first, I tried to implement a routine of creating test cases for each user story, organized per sprint. However, I soon saw that all test cases and user stories ended up 1-to-1, and the test case descriptions was just a repetition of the user story and its details, which are very straight-forward and not subject to debate or interpretation (see above). A bunch of test cases with one or two steps that were 100% obvious when looking at the User Story felt like a waste of time. So I/we decided to try to skip the Test Cases, and just use the comments and status of the user stories, since the test cases have proved to be nothing more than extra work so far. With only two weeks per sprint, doing double or unnecessary work/documentation is not desirable. There is also no "higher authority" requesting test status in a way that requires test suites and test cases with specific metrics.

Any thoughts on this? I'm used to PBIs in a Scrum setting, which were often complex enough that they needed a least a test case with a lot of detailed steps, and sometimes more than one. Or there were quite big Test Cases that naturally covered more than one PBI. In this case, that hasn't been the case yet. In this setting, the test cases are so very small and isolated, and quick to test. (The quality delivered for each User Story is another story altogether, but that's a work in progress with the developers and leads.)

I've been thinking about the need for test cases for the FEATURES, which I think is more customer-related. However, these are just a collection of user stories that are already tested. Their titles aren't more than short description either. But I would guess that this is the place for test cases, since I think they should be tested or verified by the customers/users of the system.

I guess what I'm asking is suggestions on what/how to test what/where, and how to organize it with regards to documentation, tracability and most of the common issues related to testing. There is no real involvement of customers/users yet, but I'm thinking that this needs to be done, at least on a FEATURE or EPIC level. But this requires the time and knowledge to write test cases based on value chains and scenarios, which I think is pretty different than from a system/User Story viewpoint.

It's worth to note that this is a permanent development team working on a system permanently, and now working on version 3 of the same system. So the goal is the same system, just with more and better functionality, as well as added features requested both from external customers and internal users. But as of now there's no one to act as a product owner or requirements responsible. This is also in the pipeline.

Answer 1

There is nothing scum(ish) about this project

• Team members are not involved in deciding what can be accomplished in the sprint. No sprint planning. It is top-down.
• No coordination among team members in the form of a Daily Stand-up.
• Team is not attempting to create a potentially shippable increment at the end of the sprint.
• Team is not showing the work completed at the end of each sprint to stakeholders and seeking their feedback.
• Development team is not doing the effort estimates.
• No mention of any Scrum Master or Product Owner roles.

You have the right gut feel when you said:

There is no real involvement of customers/users yet, but I'm thinking that this needs to be done, at least on a FEATURE or EPIC level.

The biggest risk seems to be reaching the end of the project and the customer looking at it for the first time ever and saying, "This is not what we asked for".

So, follow-up on your gut feel and use the notion in the team that this is a scum(ish) project to organize Sprint Reviews with the customer at regular intervals. Based on the feedback:

1. You will know how well the user stories represent what the customer asked for.
2. You can tweak when/how to test.

And this will also help you to follow-up on your other gut feel:

...this requires the time and knowledge to write test cases based on value chains and scenarios, which I think is pretty different than from a system/User Story viewpoint.

Answer 2

My project is also in active agile development of a newly released product.

The primary purpose of my tests is to ensure that we don't accidentally break something that we have already delivered, so I focus on adding test suites for features added in each sprint.

Thus, in your situation, I wouldn't have any qualms about adding tests that match user stories even though it is obvious, because I'm not testing whether we did this sprint right: I'm adding quality control machinery to make sure it stays right.

I typically don't involve the user in testing except when they have directed us to add a new feature that breaks or changes an old feature. In this case I show them the failed tests and say "this is what you expected to happen, right?" and get their signoff before updating the tests to the new behavior.

Bottom line, the tests in my project are for the development team, not the product owner.