3

In my software, a lot of services expect user to be authenticated before performing.

In positive scenarios, I have this kind of context:

Given I am authenticated
When I create meeting X
Then meeting X is well created 

I am used to write a symmetrical scenario like this for each of my services that deals with user authentication requirement:

Given I am not authenticated  //note the 'not' word
When I create meeting X
Then meeting X is not created
And an authentication error should be thrown

It forces developer to handle check of user authentication at the top of the service implementation, to prevent any potential "hacker" or malicious code to call the API (in this example 'meeting creation') directly while not being authenticated.

Is it a practice I should keep?
Am I right to consider this "defensive" scenario as a real business rule that I may discuss with "Business" team?

Note that my acceptance tests do not test through the GUI part, but only directly through the use cases part (services / business rules).

  • 3
    Creating tags for gherkin or cucumber might be useful for questions like these, but might easily slip over the edge into engineering. – Todd A. Jacobs Jul 7 '17 at 16:25
  • "Given I am not authenticated...When I create meeting X": Something else to think about is why an unauthenticated user can even attempt to create a meeting. This seems like a UX design problem more than a useful test case. As always, YMMV. – Todd A. Jacobs Jul 22 '17 at 15:40
7

Short answer: yes, it is perfectly fine to account for negative cases.

I'm used to seeing this a bit differently. Usually a User Story is one step up like:

As a user, I'd like to be able to add a meeting on the calendar so that I can track my schedule for the day"

Then I would have both of these as acceptance criteria on that story.

This recommendation is more a matter of style though. I like this because it groups them better so that important business cases don't fall through the cracks. What you are doing is certainly not wrong.

  • Seems that your sentence is the label of a feature for me, not of a user story. The feature aims to focus on the goal ("I'd like to..."), a user story (scenario) is a path to achieve (or not) it with "Given/When/Then". – Mik378 Jul 7 '17 at 19:24
  • 1
    I've definitely had a different experience in terminology, but it sounds like we're on the same page - the "I'd like to" describes the goal and the "Given/When/Then" describes the expected behavior inside of there. Either way, I think it's both good and even important that you are accounting for expected negative behavior. I expect that your testers will uncover more still that need to be accounted for. – Daniel Jul 7 '17 at 20:19
4

Capture Business Concepts (Not Engineering Steps) In Gherkin

Is it a practice I should keep?

Maybe. Negative test cases, like boundary conditions, are good things to test from a quality assurance perspective. However, from an engineering or product management perspective, it's worth asking why you need a narrow scenario like this rather than either a more comprehensive feature or a scenario outline. And from a business perspective, low-level details in acceptance tests are generally an anti-pattern.

Express the Business Case

For example, rather than having all sorts of odd entry points as separate use cases, a more comprehensive feature might say:

Given that a user is not authenticated
When the user performs any action
Then the user is redirected to the login page.

This captures the real business case better and more clearly, although you would probably want individual steps or a scenario outline to test all the possible entry points and to be able to report individual failures more effectively.

The benefit of this type of story is that it captures the business logic a lot better than an implementation-centric story does. On the other hand, a failure within one of the steps is a lot less communicative, and requires knowledge of the underlying steps to narrow down specific failures.

There's a middle ground: senario outlines.

Use Scenario Outlines

The main problem with expressing only the business domain without also defining some context or targeted use cases is that you can't determine with specificity where a test group is failing. That's where Cucumber scenario outlines can help.

For example:

Scenario Outline: force authentication on all pages
  Given an unauthenticated user
  When the user connected to the <function> page
  Then the user should be redirected to the login page.

  Examples:
    | function |
    | meeting  |
    | calendar |
    | profile  |
    | launch thermonuclear warheads |

With this type of outline the business goal is clear, and you also have a set of high-level tests that can pass or fail independently of each other. The top-level reporting is both granular and cohesive, while avoiding the sort of "test sprawl" that you would get from adding negative scenarios for each behavior.

Summary

Testing is more of an art form than a science, so your mileage may vary. However, agile testing should function as self-documenting behavior rather than low-level detail, and acceptance criteria should focus more on business logic or user-visible behavior than on underlying implementation details.

  • what is different from my practice in my OP? The fact that I mention the return of an error? sounds too technical even if "abstract" ? I need it because my acceptance tests target use cases directly, not GUI or any "redirection" mechanism. – Mik378 Jul 7 '17 at 18:46
  • 1
    The difference, if I understand it correctly -- at least I would vote for that --, is that you maintain this general requirement in a single place. This simplifies maintenance of requirements (imagine what happens if there is a change regarding this; in your case, you'll have to update many places) and also derivation of test cases or low level technical requirements. – loom with a crew Jul 9 '17 at 10:04
  • @Mik378 You're inherently trying to raise steps to first class status in your scenarios. While you can do this, and the merits of doing so could be argued elsewhere (such as on a site about QA test engineering), this is a site about project management. From a PM perspective, low-level tests and engineering-oriented language shouldn't be first-class project artifacts. In fact, I would consider engineering-level reporting masquerading as "acceptance testing" a serious project smell that should definitely be treated as an anti-pattern. YMMV. – Todd A. Jacobs Jul 10 '17 at 16:33
  • @CodeGnome I don't think they are low-level tests. A scenario isn't a low-level test but a spec demanded by the Business. – Mik378 Jul 10 '17 at 16:45
3

TL;DR: Yes, you may keep it this way.

Long Answer

Depending on what you are using the scenarios or business rules for, there are several ways to write them down

First case: The scenario is used for acceptance criteria

Story

As a user, I want to create meetings, to...

Acceptance criteria

  • Authenticated user may create meetings over a GUI / API
  • Anonymous users should not be able to create meetings and get an error instead

First case: The scenario is used for acceptance / regression tests, which are fragmented instructions to click through the application. All these tests should pass after every story (or at least before every release).

Incomplete example:

happy path

  1. open browser -> login screen appears
  2. authenticate -> login successful message
  3. create a meeting -> see meeting details

without authentication

  1. open browser -> login screen appears
  2. create a meeting -> see creation failed due to authentication error

Side note

I would recommend to do the regression tests through the GUI by a user. If they are written well, they can be implemented in code and run automatically.

  • What you are calling story, I call it feature. What you are calling criteria, I call it a story or scenario. Those are gherkin terms. – Mik378 Jul 7 '17 at 8:55
  • 1
    I see, maybe you replace a tag with gherkin :) Nevermind, besides the terms, is this answer somehow useful? – ppasler Jul 7 '17 at 8:59
  • Yes, the answer is useful but I disagree about the fact that regressions test should happen through the GUI. 8thlight.com/blog/doug-bradbury/2011/04/26/… (at the end of this article) – Mik378 Jul 7 '17 at 9:08
  • Interesting thoughts :) Just to get this straight, besides regression tests, we are using unit tests, to get fast results (almost no waiting) and automated regression tests every day, checking all acceptance tests on "real" environments (plus several version and DB systems). This way we make sure everythings works as it should, even for small changes. Before that we often had the situation, that too much in our tests was mocked away and gave errors after releasing to other environments. – ppasler Jul 7 '17 at 9:16
  • I practice ATDD, consisting in acceptance tests being implemented through some TDD cycles (unit tests). As soon as those unit tests pass (incremental algorithm), the concerned acceptance test should pass. – Mik378 Jul 7 '17 at 9:17
1

As others have written, accounting for negative cases is good. However, the structure of your negative case is not, because it contains an internal contradiction.

The problem is that you are describing outcomes as if they were user actions. But they are actually the action the software takes in response to the user action. Failures have user actions, but the expected output never occurs, because the failure prevents it from actually happening, so a story predicated on the event is never satisfied.

Bad:

Given I am authenticated

When I create meeting X

Better:

Given I am authenticated

When I submit a "create meeting" request

This is still operating at the level of the backend, it doesn't refer to specific UI elements such as menus or buttons, and covers network activity not generated by the approved frontend (covering the "hack" concern).

  • hmm, I disagree like I explained above. Acceptance tests should not focus on the GUI or controller part but on the use case directly. When you mention "submit"/"request", those are terms of web => gui. Gui is an implementation detail, and details should not be part of acceptance tests. For instance, if your client is a pure console (terminal) rather than web, your acceptance test's terms would be inappropriate. – Mik378 Jul 7 '17 at 19:14
  • I thought about your point. I found the following example that pretty improves the coherence of a scenario: agilealliance.org/glossary/gwt (at the bottom). I like the approach "I attempt to...", meaning that the output is not predictable yet before acting. It is enough abstract to fulfill my UI agnosticism and is similar to your "I submit ...". – Mik378 Jul 11 '17 at 20:29
  • 1
    @Mik378: Yes, that should work out well too. – Ben Voigt Jul 11 '17 at 21:01

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.