QA phase reporting for SW development

Question

In our project, we have a testbook with some test scenarios to be performed by a QA team. Each scenario must be tested on multiple device/OS/browser (in this particular case, just 3 combinations), and for each scenario/target we have multiple statuses, for instance:

Scenario	Status - iOS/Safari	Status - Android/Chrome	Status - Win/Chrome
Click on the "Bold" button	Todo	Todo	Todo
Surfing in the USA	Passed	Passed	Blocked
Hit the road Jack	Failed	Failed	Todo
Test world peace feature	Todo	Passed	Todo

Which are the possible way to report to mid-management and program manager the overall status, periodically? The data to show are:

• count of test scenario in the testbook
• count of executed test scenario
• success rate for each target device/os/browser, based on the number of executed scenario to date

The only thing that comes to my mind (avoiding a simple table) is the stacked bar chart (one bar for each target device/os/browser). Is there anything clearer?

Answer 1

One challenge is that there is no "right" answer. But this is how I'd get to a "good" answer:

The real problem you're tackling is that middle managers have reason to want both the at-a-glance view that the people above them need and the specific view that team members need. You said you handle the exec summary view fine, so I'll leave that aside.

For the second, I would look at it broader. Many people need this view. I expect many QA team members need this at a glance. Those test cases may be separate, but the system they are testing isn't. Knowing that Hit-the-road jack failed on everything but the bold button only fails on Safari is important information. Similarly, your devs want this same context. That is to say nothing of planning time for testing and working bugs that come out of it. Now, once you've solved that problem, you can just give the middle manager access to that view and your problem is solved.

Answer 2

I think that this level of reporting would not really be meaningful outside of a team-operations level. Instead, I would try to attach each test or group of tests, both to a particular "aspect" of the product, and in this case to a particular "platform." And, I would try to attach to each test some indication of actual severity: green-light, yellow-light, red-light, thermonuclear explosion. Maybe also "flashing yellow" if something is blocking us from having results yet.

Now, tell me the status of each "aspect" of the product on each "platform." And, some assessment of the "severity" of each square in that grid.

Testbooks necessarily emphasize coverage. So, a testbook might contain many different almost-redundant tests – "wheel #1 is round" and so-forth. But this probably means that "the number of" tests is not particularly interesting, or management-useful.