How much automated test coverage should I aim for in an agile project?

Question

I have generally used manual testing in my projects. In the proposal plan, I am going to introduce automated testing this time.

Few things I am going to add:

1. Unit Testing using Junit Test cases
2. Continuous Integration using Hudson or Cruise Control StackExchangeLinkIReferred

For Integration and System Testing, how much automation I can do? My project is a web project. JSP->Spring->Hibernate->Oracle

It seems there are some tools like FitNesse which are useful for agile purposes.

As per my understanding, for screen capture, we should not be using automation as screen often changes but other areas can be fully automated even if there are additions in each sprint by tools like FitNesse.

Can you please share your suggestions?

Answer 1

You should definitely not try to write "as much automation as possible" or fall into the common trap of thinking that 100% automation is best and should be aimed for.

Writing automated tests takes time, running them takes time and maintaining them takes time. If only 40% of our tests are useful, you wasted time writing to remaining 60%. You also need to wait for the other 60% to finish executing to get the results of the useful 40%, and have to spend time maintaining them.

So, what should you test through automation? Here are some things to consider

1. Core functionality of your app. If your app is a website monitoring tool then this would probably be any alerts which need to fire if a site has problems. If these go wrong your app isn't serving it's core purpose so having automated regression tests to run on every code change makes sense.
2. The basics (often called a smoke test). Ie using selenium to click all the links to navigate through a web app and make sure the server responds with a 200 for each. Note, I don't mean checking the content of the pages, just that you can navigate the app and it's not obviously broken. These tests ensure any manual testers aren't having their time (and morale) wasted with obviously broken builds.
3. Anything fragile which is likely to break. This is hard to define for new code, but if you're working on some legacy code which you know to be bug prone then building tests around any changes you make is a good way to build towards being able to refactor more safely.
4. Things the developers wanted to automate because it was easy to do. Sometimes it makes sense to add tests while you're building something, that shouldn't be discouraged.

I'd suggest tests are structured along the above lines, in that way when one fails people know why it was created. If it is the test that is wrong and not the code under test they should understand why it's important to fix the test rather than delete it.

You can obviously adapt the above as you learn more and make decisions as to what code is worth writing tests for.

Some other thoughts

1. Do consider UI testing, but as others have mentioned try to remove dependency on the structure of the UI where possible.
2. Get set up with Jenkins (or similar) asap and start having all tests run on commit
3. Try to keep the test run time in Jenkins low. This has two important effects. Firstly, developers are more likely to get notified of a failure before they've got too far into the next piece of code. Secondly, changes from two or more developers are less likely to be part of the same build so it's more obvious who is responsible for fixing them. Splitting tests up into groups to run in parallel seems to be the simplest way to achieve this.

Update: I wrote up some thoughts on this here http://blog.getcorrello.com/2015/11/20/how-much-automated-testing-is-enough-automated-testing/

Answer 2

You should aim for as much automation as possible. Automated tests finish faster than manual tests, and therefore you can significantly reduce idle time; when you are waiting for test results. Of course, it matters what kind of tests you automate and how much you write of the kinds; we call this approach the test pyramid:

The width of a layer tells you the amount of [automated] tests you ought to write.

You should put your GUI related tests to the top and it is worth the effort. Screen capture and comparison is very useful, until you don't write more than 10 tests for them (2 for each browser). Automate this one, too.

Agile projects also use manual testing for finding more test cases; we call it exploratory testing.

You handle the system as a black box, and you different scenarios manually until you find a new test case. When you have found it, you write a test case on the proper layer in the test pyramid. Not all tests should be on the GUI level despite the fact that you found it there. For example, you don't need a GUI test for a specific error case if you can test reproduce and test it on the unit test level.

You should forget about FitNesse. Cucumber is a way better alternative. I wrote a nice tutorial about it a while back.

For about cruise control, too.

Answer 3

Building automated tests is going to take time and effort. But it will also save time and effort in manual testing and may reduce the need for rework.

The trick is to find the right balance for your team. There are two approaches you might want to consider:

The first approach is to analyse your past bugs and try and target your automated tests at the components and functionality that fails the most often. This approach can provide the maximum benefit for the smallest investment of time and effort.

Another approach is to work with the stakeholders to determine which functionality is most important in the product. Add your automated coverage starting fom the most important feature and working your way down the list. I have worked with some teams where they submit the list of automated tests to the Product Owner for approval. The Product Owners may then say something like "that's great, the functionality you are covering is all the important features of the applications. Problems elsewhere we can live with if necessary".

One piece of advice for web based automated testing is to make your tests as independent as possible from the page layout. For example you can use IDs to identify the elements on the page and these IDs can remain constant even if the layout changes.

FitNesse is a good tool as it allows for a lot of exploratory testing by testers or non-technical stakeholders. The idea with FitNesse is to separate the testing code from the test scenarios. As an example, say your team was working on a calculator application. The developer would write some code for adding together two numbers. They would also write a FitNesse 'test fixture' which passed numbers to the adding code and returned the answer. FitNesse then provides an interface for the tester (or even business users) to pass data to the test. So the tester might pass 2 + 2 and expect an answer of 4. One benefit of this approach is that the testers can focus on test scenarios and not worry about how to write the automated tests.

Cucumber is a tool that is used to do Behaviour Driven Development (BDD). With Cucumber you tie your requirements closely to your tests. Cucumber allows the team to write the requirements using business friendly language. Then Cucumber provides the hooks between these requirements and testing code. Using the calculator example again, you could write a scenario in Cucumber like "given I have two numbers (5, 7) when I add those numbers together then I will get a sum of 12". Cucumber would then link this scenario to an empty code method. It would be up to the developers/testers to write the code that made the test work.

You can see how these kinds of testing tools give you a neat way to integrate automated tests in to your work. But take note they don't actually include the test automation itself. I would suggest you try them out and see which ones your team finds to be useful.