10

I have generally used manual testing in my projects. In the proposal plan, I am going to introduce automated testing this time.

Few things I am going to add:

  1. Unit Testing using Junit Test cases
  2. Continuous Integration using Hudson or Cruise Control StackExchangeLinkIReferred

For Integration and System Testing, how much automation I can do? My project is a web project. JSP->Spring->Hibernate->Oracle

It seems there are some tools like FitNesse which are useful for agile purposes.

As per my understanding, for screen capture, we should not be using automation as screen often changes but other areas can be fully automated even if there are additions in each sprint by tools like FitNesse.

Can you please share your suggestions?

7

You should definitely not try to write "as much automation as possible" or fall into the common trap of thinking that 100% automation is best and should be aimed for.

Writing automated tests takes time, running them takes time and maintaining them takes time. If only 40% of our tests are useful, you wasted time writing to remaining 60%. You also need to wait for the other 60% to finish executing to get the results of the useful 40%, and have to spend time maintaining them.

So, what should you test through automation? Here are some things to consider

  1. Core functionality of your app. If your app is a website monitoring tool then this would probably be any alerts which need to fire if a site has problems. If these go wrong your app isn't serving it's core purpose so having automated regression tests to run on every code change makes sense.
  2. The basics (often called a smoke test). Ie using selenium to click all the links to navigate through a web app and make sure the server responds with a 200 for each. Note, I don't mean checking the content of the pages, just that you can navigate the app and it's not obviously broken. These tests ensure any manual testers aren't having their time (and morale) wasted with obviously broken builds.
  3. Anything fragile which is likely to break. This is hard to define for new code, but if you're working on some legacy code which you know to be bug prone then building tests around any changes you make is a good way to build towards being able to refactor more safely.
  4. Things the developers wanted to automate because it was easy to do. Sometimes it makes sense to add tests while you're building something, that shouldn't be discouraged.

I'd suggest tests are structured along the above lines, in that way when one fails people know why it was created. If it is the test that is wrong and not the code under test they should understand why it's important to fix the test rather than delete it.

You can obviously adapt the above as you learn more and make decisions as to what code is worth writing tests for.

Some other thoughts

  1. Do consider UI testing, but as others have mentioned try to remove dependency on the structure of the UI where possible.
  2. Get set up with Jenkins (or similar) asap and start having all tests run on commit
  3. Try to keep the test run time in Jenkins low. This has two important effects. Firstly, developers are more likely to get notified of a failure before they've got too far into the next piece of code. Secondly, changes from two or more developers are less likely to be part of the same build so it's more obvious who is responsible for fixing them. Splitting tests up into groups to run in parallel seems to be the simplest way to achieve this.

Update: I wrote up some thoughts on this here http://blog.getcorrello.com/2015/11/20/how-much-automated-testing-is-enough-automated-testing/

  • It's refreshing (and reassuring) to hear someone promoting a more pragmatic approach to test coverage, rather than the "always 100%" dogma. – Michael Oct 13 '15 at 11:18
  • @Robin, your points make sense but in agile development and testing are done together.Continuous Integration -producing a new build every time a programmer completes a work on a User Story. These builds include regression & integration testing to ensure the quality of the evolving product. If the TDD & integration tests are not automated, Continuous Integration is infeasible and we are unable to utilize it after the completion of each story? Without automation,we cant run all the integration/regression tests after each story in such short time frame.what are your views on this – Dimple Sahani Oct 16 '15 at 6:24
  • @DimpleSahani I don't think my answer precludes you doing dev and testing together, and I was in fact assuming that was the case. If a test needs to be run as part of the integration/regression test suite after every change then it should ideally be part of your automated testing. The question I was raising, is what actually needs to be tested every time? I don't believe that is 100% of your application functionality. All I am saying is look at what tests you really need automating and spend the time to automate those. Manual testing is fine when it doesn't need to be done for every change. – Robin Oct 16 '15 at 10:51
  • @Robin , sorry I didn't write my point properly...what I meant was that - 1 or 1.5 year down the line, my project will become quite big and if I have the automation of only "smoke testing" or "imp functionality" , doing manual integration/regression testing along with each story will not be feasible as sprint is just 2-3 weeks cycle.I agree 100% is not possible at all and to start with we need to look at what tests we really need automating and in the beginning spend the time to automate only these. May be,Later on, we can build more test cases. – Dimple Sahani Oct 16 '15 at 13:24
  • @DimpleSahani I think you misunderstand my point. To be clear, if a test is worth writing, including if it will save time performing regression tests in two years time, you should write it. I think you will get the vast majority of the value from coverage of less than 100% of your code base. But, if you wouldn't be happy having a some piece of code not tested for regression bugs then you ideally need to automate those tests. Also, I'd advise against the idea that you can write more tests later on. This means you are taking on significant technical debt which will be hard to pay off later. – Robin Oct 16 '15 at 14:26
2

You should aim for as much automation as possible. Automated tests finish faster than manual tests, and therefore you can significantly reduce idle time; when you are waiting for test results. Of course, it matters what kind of tests you automate and how much you write of the kinds; we call this approach the test pyramid:

test pyramid

The width of a layer tells you the amount of [automated] tests you ought to write.

You should put your GUI related tests to the top and it is worth the effort. Screen capture and comparison is very useful, until you don't write more than 10 tests for them (2 for each browser). Automate this one, too.

Agile projects also use manual testing for finding more test cases; we call it exploratory testing.

You handle the system as a black box, and you different scenarios manually until you find a new test case. When you have found it, you write a test case on the proper layer in the test pyramid. Not all tests should be on the GUI level despite the fact that you found it there. For example, you don't need a GUI test for a specific error case if you can test reproduce and test it on the unit test level.

You should forget about FitNesse. Cucumber is a way better alternative. I wrote a nice tutorial about it a while back.

For about cruise control, too.

  • Zsolt, your comment is very useful...But I few queries 1)as the width of layer in pyramid shows that we are not able to achieve the 100% automation in integration...is it because of the nature of agile projects that it has ever changing needs...if that is the case it effects lowermost layer also but that it is bigger...so I m not able to get why so? 2)I will go thru Cucumber article that u have suggested cause I m confused with when shd we go with screen capturing testing tools vs programming based testing tools 3)You have written cruise control article also, please share link – Dimple Sahani Oct 11 '15 at 9:28
  • 1. In agile, when the project changes, you changes the tests. In TDD fashion you change the tests first. It is very expensive to reach 100% coverage - automated or manual. People usually aim for 80%. The last 20% is very expensive (twice as much as reaching the %80). – Zsolt Oct 11 '15 at 9:36
  • 2. you'll write a test that captures the screen and compares the reference to the current. This is the industry practice. – Zsolt Oct 11 '15 at 9:37
  • 3. Nope, cruisecontrol is very old and rarely used, so is Hudson. Go for jenkins. – Zsolt Oct 11 '15 at 9:38
  • Zsolt, could you please guide why you feel that cucumber is better than FitNesse. I have read some basic overview articles of both and I could not figure out the reasons of using any particular one.Also , there are some other tools like Jbehave and concordion. – Dimple Sahani Oct 12 '15 at 14:12
2

Building automated tests is going to take time and effort. But it will also save time and effort in manual testing and may reduce the need for rework.

The trick is to find the right balance for your team. There are two approaches you might want to consider:

The first approach is to analyse your past bugs and try and target your automated tests at the components and functionality that fails the most often. This approach can provide the maximum benefit for the smallest investment of time and effort.

Another approach is to work with the stakeholders to determine which functionality is most important in the product. Add your automated coverage starting fom the most important feature and working your way down the list. I have worked with some teams where they submit the list of automated tests to the Product Owner for approval. The Product Owners may then say something like "that's great, the functionality you are covering is all the important features of the applications. Problems elsewhere we can live with if necessary".

One piece of advice for web based automated testing is to make your tests as independent as possible from the page layout. For example you can use IDs to identify the elements on the page and these IDs can remain constant even if the layout changes.

FitNesse is a good tool as it allows for a lot of exploratory testing by testers or non-technical stakeholders. The idea with FitNesse is to separate the testing code from the test scenarios. As an example, say your team was working on a calculator application. The developer would write some code for adding together two numbers. They would also write a FitNesse 'test fixture' which passed numbers to the adding code and returned the answer. FitNesse then provides an interface for the tester (or even business users) to pass data to the test. So the tester might pass 2 + 2 and expect an answer of 4. One benefit of this approach is that the testers can focus on test scenarios and not worry about how to write the automated tests.

Cucumber is a tool that is used to do Behaviour Driven Development (BDD). With Cucumber you tie your requirements closely to your tests. Cucumber allows the team to write the requirements using business friendly language. Then Cucumber provides the hooks between these requirements and testing code. Using the calculator example again, you could write a scenario in Cucumber like "given I have two numbers (5, 7) when I add those numbers together then I will get a sum of 12". Cucumber would then link this scenario to an empty code method. It would be up to the developers/testers to write the code that made the test work.

You can see how these kinds of testing tools give you a neat way to integrate automated tests in to your work. But take note they don't actually include the test automation itself. I would suggest you try them out and see which ones your team finds to be useful.

  • Barnaby Golden,it seem you have worked in FitNesse... I have read some basic overview articles of cucumber and Fitnesse and I could not figure out the reasons of using any particular one.Also , there are some other tools like Jbehave and concordion...can you suggest some scenarios if you have some practical example. – Dimple Sahani Oct 12 '15 at 14:14
  • I updated my answer to include more detail on FitNesse and Cucumber. I'm afraid I don't have much experience on JBehave or Concordion. – Barnaby Golden Oct 12 '15 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.