Testing and customer acceptance of a project is dragging on forever

Question

I'm the only software developer and project manager for the technical side of a project we're delivering for one of our clients. We initially agreed to and signed-off on a high-level spec that we would deliver whatever would make them happy.

We delivered the product (code) to them about a month ago. We internally test this product ourselves (~150 unit tests in JUnit, including simulations of end-to-end user-style test.) We asked the customer to test it and sign off.

They identified a small problem. And then another. And a third. More and more issues are emerging. Some of them are simple to diagnose and fix; some of them are due to an incomplete and vague spec delivered to us; some of them are feature requests for new functionality; some of them are technical or architectural holes; some of them are due to holes in the design.

Everybody is getting antsy. Our company needs to get this project done ASAP, because it's holding up other, mission-critical work for us. The customer doesn't understand why testing and resolving issues is taking forever.

What's the best way to resolve this situation?

We asked the customer to identify whatever tests they're running, in as complete detail as possible, so we can see what they're testing and make that work. However, their executive sponsor is getting antsy. This was supposed to be a small, quick project which is now dragging.

Edit: In an attempt to clarify exactly what they want and what we need to deliver, we created a "user acceptance testing" doc with test-cases that they're running. So far, it covers around twelve tests we've added.

The problem is, they add tests that are not tests (eg. "Department X will test the deliverable") or they keep adding more specific requirements (eg. how fields will look) to the testing. It's too open-ended, perhaps, for us to clamp it down to control scope.

Answer 1

It's in both parties interest for this project to be finalised. The best course of action is to facilitate a meeting with all key stakeholders, to agree on a strategy which will enable the completion of the project within a agreed specified period.

You will need to establish an understanding and agreement with the customer of

• Customer testing plan.
• Definition bugs, you should only be fixing bugs that affect the agreed deliverables in the original specification (prioritising is important here). Any bugs outside these parameters should be defined as after release issues.
• Test cycle that has been identified by the customer, including duration.
• The SLA after implementation, if any. Come to an agreement what issues (either low priority or out of scope) will be investigated further.

Answer 2

It sounds a bit like you have lost control of the process. Someone needs to own the testing record/log.

Go back to the requirements and list the use cases/processes. then list the tests for each. Putting columns for who will test will help illustrate the depth of the involvement.

Then get the people together and get agreement on the scope of testing. Then identify what has already passed and what still needs to be tested.

Talk to the sponsor about setting a deadline.

Often this happens when people are afraid to put their name on the pass because they anticipate being blamed for anything that happens.

And, next time. Get agreement way ahead of testing how that will work.

Answer 3

some of them are feature requests for new functionality;

Push back on every single bit of new functionality until the existing bugs are done.

Some of them are simple to diagnose and fix; some of them are due to an incomplete and vague spec delivered to us; ... some of them are technical or architectural holes; some of them are due to holes in the design.

This part appears to be a severe infection of I'll know it when I see it syndrome. You're a lot closer to the ground than I am, but here are a number of possibilities:

• The subject matter expert doesn't know how to explain what they want, so they don't explain it and can't be more articulate than "I think that couch would look better on the other side of the room."

• The subject matter expert feels threatened, or has a long history of keeping things close/confidential so that they don't get replaced. This is a strong indicator when specs turn into "department x will test..."

• The client doesn't want things clarified and written down because that has legal ramifications - you could go to court and say "we did what they asked us" but the death of a thousand change orders that you're encountering is a countermove.

• The client is disorganized.

Department X will test the deliverable

No. Push this right back. It needs to say how department X will test it. What user cases? What scenarios?

they keep adding more specific requirements (eg. how fields will look)

Push back on this as well. Defer all prettiness features like this until the next version. Otherwise you'll end up spending the next 6 years arguing about which Pantone color that all submit buttons should be. If you are billing for these "how many angels can dance on the head of a pin" discussion, then feel free to let it continue, otherwise cut them off.

However, if the client is obsessively visual (do they get bent when you move something on their desk, or touch their monitor, must everything be positioned 'just so' before they can start work, or they're concerned about how folks dress), then getting the stuff to line up and be the right font is more important to them than if the darned thing actually does the job it is needed to do. If this is the sort of person they are, then prettying up the application is your number one task.

We asked the customer to identify whatever tests they're running, in as complete detail as possible

I've had this with a project that dragged on forever. The key is to sit down with the subject matter expert and go through the workflow. Be prepared to print out tons of screenshots and write notes on every screenshot. If you can record the conversation do so, and write stuff up later. This is the only way to make things work with an SME who won't part with details, or in the situation I just got out of, the SME had been secretive for so long that now that they were within 6 months of retirement (they're a federal civil-service employee, so it isn't like they could be fired), they still couldn't open up.

If you haven't done so already, set up something for tracking bugs and feature requests. Mantis, Trac, whatever. Take an evening to write up everything in a list and triage it into must do , should do and won't do (more politically, call this last category "deferred"). All new features go into "deferred" until the project has passed acceptance testing - even if it is one line of code.

We initially agreed to and signed-off on a high-level spec that we would deliver whatever would make them happy.

Really? They want a pony because that will make them happy, and you'd get them one?

Answer 4

Don't let them push scope creep or change requests into the testing process.

Clearly delineate what is a bug fix vs what is new functionality / scope creep.

Push back on all scope creep and set a timeline for getting in/reporting all bugs, after which you will walk away from the project.

Answer 5

I agree with Perry, Dave and Fun Mun. You have to get back to them with your high level requirements, show them how you've built it and your testplan and go through it requirement by requirement.

Write down everything they say, find agreement on which things should be changed, what are real bugs and what is out of scope. Then determine a warranty period for bugfixing and draw up a new agreement.

Take your boss with you. If he wants this customer to be happy, he'll have to work hard too.

Unless you guys are good negotiators, this is probably going to cost you. Then it becomes a commercial/business decision.

Good luck!

Answer 6

Next time specify a maximum acceptance time, such as 3 months. At least then you can get paid, even if the work drags on.

Answer 7

Try to go back to the drawing board and get the specification detailed out. With a proper specification, you will have negotiation power.

A specification will allow you to analyse the gap between the truth and what you have. You can use the gap analysis to discuss with all the stakeholders and attempt to reach a decision. It allows you to let the users prioritize on what they want. You can say, "This is what you have requested. It is going to take 2 million years to complete all of them. Tell me what you want done and I will add time to the timeline."

This gives you the rights to say "no" to ridiculous requests.

It also offers your the chance to say, "hey, I think you have a great idea there, and best of all, it can be applied to a lot of other modules/places. Why don't we keep it for the next version so that we can put it in more modules/places."

If a specification can't be obtained, whether through resistance or sheer ignorance of the users, then you may want to put up a list of TODO items for everyone to see. Have a meeting and let them vote for which goes in and which goes out. Give them a choice of "A" or "B"? Don't answer "do you want this?" The answer will always be yes. And UAT is exactly the "do you want this" type of question.

Answer 8

It's never a good idea to sign off on such a vague specification. By signing a document that says you'll do whatever it takes to make them happy, you effectively give all your bargaining power away to the client.

It's like indentured servitude, but worse!

They say the customer is always right, but that's not the case in engineering. In engineering, the customer and vendor are two negotiating parties participating in a give and take relationship.

As a vendor, it's your job to only sign off on objective, measurable, criteria. Without objective, measurable criteria, there is no recourse for resolving disputes that arise in the project with regards to scope. A good specification can douse the flames of a potentially heated confrontation and give both parties a point of reference for determining scope boundaries.

The best medicine for a situation such as yours is prevention. This is a great lesson learned for your organization, and this will make you stronger.

To fix this, you'll have to draw a line in the sand. Meet with stakeholders to define clear, objective testing criteria and clear, objective acceptance tests. Your only bargaining chip at this point is the clock. There will be a point where the client won't want to drag this on any longer too, and you can use estimates of scope creep to reduce that scope. Eventually, it will cost the client money too if they can't use their software, and they'll have to pay you something if they want to ever use it.

Remember, you choose your clients just as much as they choose you. If your client can't agree on objective criteria that you can measure, then you don't want to do business with them. It's your job to teach your clients to be good clients by taking control of the situation and only agreeing to work on tasks that have a clear, objective, measurable answer. If they let you be the expert you are, then they are worth doing business with.

Answer 9

you can say that let us move ahead with what ever is coded and we can take up additional requirements/improvements later as part of enhancement project or maintenance work. This will ensure that you deliver what has been agreed in the original requirement specs. this also means that you will get paid for the maintenance work which will follow. Good luck with such customers