Acceptance Testing fails on owners machines only

Question

I'm in the process of completing the acceptance testing with a customer for a web based system and after 3 rounds of acceptance testing we have found that one particular feature of the system does for some reason not work on the clients 2 machines we have used for testing.

It is pretty unusual for a browser based application and we have gone through an in-detail compare of the computers and the test environment. There are also possible programming problems that may exascerbate the problem

Unfortunately a lot of other things have gone wrong in the project as well, and this has contributed to a loss of trust and made acceptance testing very tedious.

I'm looking for ways to manage this better in the future, apart from of course identifying the actual difference that is causing this.

Can people here recommend project management and contractual preparations that are realistic for this scenario? The applications target market is only about 100 people with the 2 key people at the client obviously being super admins of the system.

• my business partner thinks its too hard a requirement to force acceptance testing as complete if only tested on a platform like browserstack.com - what do other people think
• should I include the right to force a memory test or other IT checks on the desgnated test equipment?
• or is this simply something that can happen, bad luck?

Answer 1

Actually I think you see this more with web applications vs 'normal' apps. Clients can have a whole host of browsers/plugins/firewalls etc which you dont have in your test enviroment.

In regards to how to mange these problems, it sounds like you are doing the right thing by testing on the customers own devices.

I would add that carefull and conservative choice of acceptance criteria is a must.

Ie. Always specify supported browsers, stay clear of low response times, state requirements such as good internet connnection, no plugins, open firewall etc.

Another good thing is early testing with the customer. Get a basic site up and make sure they can see it. Do frequent mini demos when you are on site and catch problems with features when they are still 'new and not completely finished' so your customer will excuse and expect minor issues.

Answer 2

I'm a big fan of using Selenium Grid. It is open source and so not expensive to use.

This allows you to run lots of combinations of browsers and operating systems. You write the test once, but run it on a lot of different environments.

There is an up-front investment of time to get the grid up and running. But once that is done, the effort to maintain it is light. You will need some serious computing hardware available though, if you plan to do a lot of testing across a wide range of environments. It works particularly well if you have a virtual machine setup available as you can spin-up VMs as and when you need them for the testing.

The beauty of this approach is that it is not difficult to add in a new test environment. Take the example of the failure on your owner's machine. As soon as a problem was reported you can find out what version of browser they are using, what operating system and what plugins they have. Configure that on a grid node and add it to the testing.