Possible to salvage failing outsourced project (9 months in)?

Question

We began working with an outsourcing company in India 9 months ago on a large web project. Despite lots of rough edges, it seemed like they were making good progress in terms of completing tasks, and they kept assuring us that everything would be cleaned up by the completion deadline. The project was originally scheduled for 6 months, but now 9 months later they're still working on it. The main issue we're having is that the coding seems to have been done haphazardly/sloppily by people who were advertised as "senior developers" to us, but in reality, probably have an entry level skillset.

A small sampling of the many, many, problems we're running into:

• Pages frequently have random errors.
• When "Save" buttons are clicked, sometimes a few fields won't have their data saved.
• Basic, industry standard functionality doesn't work appropriately. Ex: There's a situation when the full login page is loaded within a placeholder on a parent page - lots of things like this.
• Outward facing pages as well as code are littered with spelling mistakes and naming inconsistencies.
• When you navigate to a page that requires you to be logged in, you're redirected to a login page, but after logging in, you're not redirected back to the page you came from.
• When filling out a form, doing something like uploading a file causes a page refresh that wipes out all the user entered data.
• Etc., etc. Literally hundreds of items like this.

It feels like death from 1,000 cuts. On literally every single page there are numerous small mistakes - mistakes that are trivial on their own, but when combined, create a completely unusable project. If it were just a few dozen items, we could fix them ourselves, but it seems like the whole project is being held up on toothpicks.

We've learned a lot, and if we could go back to the beginning, there's a lot we'd do differently - things like interviewing and hand-picking developers, creating smaller, more manageable scopes for deliverables, having a better plan to pull out if necessary, etc. But it's too late for that now, and we've already paid them 2/3 of the money we agreed on.

What we've tried (or thought to try) so far:

• We've though about asking for the developers to be replaced, but I don't know how long it would take for new ones to get up to speed over the ones who've been on the project for 9 months, or if they would even be any better.

• We've created detailed lists of all the specific problems that need to be addressed, and they keep telling us they'll fix everything, but instead, they'll fix maybe 10% of the items and tell us everything is finished and working.

• We asked them to put an emphasis on quality assurance, and they responded by telling us they added two QA resources to the project, but since then, we've seen no improvement.

• We've had multiple meetings where the focus was 100% on how the pages need to be brought up to a "production-ready standard," with all code paths being thoroughly tested, all items from the requirement agreement implemented, and all documented problems resolved, making it very clear that we would not sign off on anything until this was all completed to a satisfactory level. Their response is always that they'll take care of everything, then weeks go by, and we find little was actually done to move the project forward.

Given the bad situation we're in (being in deep - 9 months - with 2/3 of the money already paid), and developers who can't seem to program a single page that doesn't have an error in almost every code path, is there anything that can be done to get the project back on track? Our approach of "here's what needs to be done, do it" and "here are the problems, fix them," doesn't seem to be working.

Answer 1

In my experience, outsourced projects only work if:

• you - as the client - stay involved in the development of the product. You don't just hand the provider something to do and expect them to deliver in 6 months or whatever. You don't just take "your paws" of the project. It's about continuous collaboration with the provider to work on your goals.
• you have confidence in the skills of the people that the service provider has assigned to work for you. That means you had interviews with them before allowing the provider to assign them to your project. Never ask a provider to deliver you something in however way they see fit. What they deliver is just as important as how they deliver. Making the "how" their own business is a bad idea.
• you need to use an iterative and incremental approach when you work with a new provider. You also need to have someone on your side that can evaluate all aspects of the deliverables, including looking under the hood. If you don't like what you see and the provider can't improve it or you then don't like the pace of improvement, you can make a full stop then, instead of waiting 9 months to find out you have a big mess on your hands.
• you have to keep in mind that this is still your project, not the project of the provider.

At the risk of putting forth a shameless plug, what you describe is what I've talked about in the outsourcing chapter of my book. I can't reproduce all of that content here, so I'll add some possible explanations to the things you mentioned in your question.

Despite lots of rough edges, it seemed like they were making good progress in terms of completing tasks, and they kept assuring us that everything would be cleaned up by the completion deadline.

You didn't actually verify! You should have checked the progress is real. If the project is advancing through the calendar, it doesn't mean that you are making good progress and that everything will be completed before the deadline. Projects are late one day at a time but things are communicated incorrectly, with delay, or intentionally incorrect until some point down the line where all of the bad news get handed to you at the same time.

The project was originally scheduled for 6 months, but now 9 months later they're still working on it.

Was this a new provider? You should have used a Time and Material contract, but it seems you went for Fixed-Price instead. Unless you trust the provider (if you worked with them before and they have a good track record, or they are highly recommended by someone you trust for the same reason of having worked with them before) then you should always consider the possibility of project failure. Seems that you didn't. And as David Espina mentioned in his comment above, you might still not consider it even though the evidence is abundant this is a failure.

The main issue we're having is that the coding seems to have been done haphazardly/sloppily by people who were advertised as "senior developers" to us, but in reality, probably have an entry level skillset.

You didn't take the time to know the people that were working on your project, like I pointed out above. You considered this a black box with input from you on one side, and output from the service provider on the other side. You should always know what's going on inside the black box, who is working on your project and in what manner.

A small sampling of the many, many, problems we're running into [...]

The list of issues you then go on describing are silly mistakes that are a proof of inexperienced developers. In good outsourced projects, the problems should be mostly because of misunderstandings of the requirements or the domain, or because of communication issues. What you go on describing are technical issues. The code is badly written. Experienced developers will not make this type of errors so much, but errors from the kinds I just mentioned. It's obvious that their skills are lacking.

It feels like death from 1,000 cuts

Indeed. And patching this will need 1,000 bandages.

We've learned a lot, and if we could go back to the beginning, there's a lot we'd do differently - things like interviewing and hand-picking developers, creating smaller, more manageable scopes for deliverables, having a better plan to pull out if necessary, etc.

Consider it a lesson learned. Sometimes you pay a lot for a lesson. Make sure you use this knowledge going forward on this or other projects.

But it's too late for that now, and we've already paid them 2/3 of the money we agreed on.

So instead of loosing two thirds of the money you want to lose all of the money? Do you absolutely think this can be salvaged and worth putting even more money into it? Or as David Espina mentioned, are you sure you are not falling pray to the sunk cost fallacy?

We've though about asking for the developers to be replaced, but I don't know how long it would take for new ones to get up to speed over the ones who've been on the project for 9 months, or if they would even be any better.

You have no guarantee they will be better developers unless you interview them yourself and make sure they are indeed the ones working on your project. The question you should be asking yourself is why they used entry level people in the first place, and why were they advertised as seniors?

Your next points indicate that they still don't know what they are doing, because they are not fixing their issues. They seem to be buying time and making you even more invested in the project, to continue to receive payments from you.

[...] is there anything that can be done to get the project back on track?

You either:

• continue to believe them, trust them, and carry on with the project, with the risk of digging deeper and deeper into the possible loss. At this point you have to make a full stop of development and concentrate on fixing the issues. The fixing needs to be accompanied by a full set of regression tests that make sure that once they fix something they stay fixed and don't get broken again by further work down the line. They will have to invest in automated tests that don't deliver anything for you but will give you better confidence that the thing won't completely fall apart (be aware that the same developers will have to write those tests, so their quality will be just as bad as the code they are testing). Or...
• you need to make a full stop of the project. Ask them to give you all the deliverables you agreed upon at the beginning of the project - in whatever state they are - and have someone you trust on your own side evaluate everything. This will tell you if the thing can be salvaged or if it's good for scraps. Once you evaluate and know the true extent of the damage, you will be better positioned to make a decision: cancel or continue (maybe with them or with another provider, in the same manner or iteratively).

I would have wanted to end this on an encouraging note, but from my experience, with what you are describing, things aren't salvageable.

Answer 2

There will always be a ton of opinions about the approach taken when hindsight biases take control, and those opinions will seem so obvious. A high quality lessons learned analysis would be in order but you need to avoid the post hoc ergo propter hoc conclusions.

Great pilots and great flying will bounce one in or even crash from time to time. This could have been your outsourced team's crash. Their next development effort could be stellar.

What you need to do first is stop the project completely. Stop wrench turning and stop spending. Then, review what has been delivered and see if there is any of that work product that would be useful for continued work some time in the future. Run a cogent, thoughtful post mortem. Then take this project and bang it up against your other projects and reprioritize what you and your firm want to do.

Your costs and effort to date are sunk. Walk away from it and do not consider what was spent in your reprioritization efforts. Consider this project as if it is brand new, banging its business case against other competing projects. Then march on.

Answer 3

Many good points from @bogdan.

Question from me is "how eager are you to salvage this, and what means do you have available to salvage it (i.e. mandate)?"

I have some experience dealing with software development projects, including offshore projects. If you approached me and asked me to "please salvage this project", I would travel to India, maybe with a senior developer/architect, and do what Bogdan say:

At this point you have to make a full stop on development and concentrate on fixing the issues.

Go through the items, prioritize them, get them fixed, have the senior developer/architect assist with challenges related to code quality, "recurring and weird bugs", frameworks used, etc. Work your way through the backlog, one item at a time and stabalize the system.

In practice you take control of the team and what they do. You prioritize, they deliver. And your technical assistance verifies continously.

Disclaimer: no information about the size of the project, the contract, the manpower currently involved. So if we're talking 200 developers, a different approach might be needed.

Answer 4

The fact that you have issues with so many things at once perhaps means that you aren't delivering and testing incrementally. Those multiple meetings to discuss "all requirements and documented problems" may be part of the problem - I don't believe you can run a piece of work like that. What I suggest you do instead is prioritise the defects and changes on a backlog, tackle them in one short iteration at a time, automate your regression testing and then make sure you keep moving steadily forward.

If you aren't actually "live" yet then you should think about your release planning. Keep in mind that such solutions are generally never "complete" until the day you stop using them. You should plan for an ongoing backlog of work and continuous improvement rather than perfect completion. What matters is what functionality you can deliver and when.

If you have lost confidence in the development team and assuming you own all the IP then definitely consider switching service provider.