Notwithstanding how you were supposed to get paid by the client, you need to have the best interests of your client in mind. This means you need to inform your client of your inability to resolve the issue and make a recommendation to your client to either find another solution provider or to abandon the solution because maybe there might not be an available solution invented, yet. The desire to continue chasing sunk costs is huge; this is a commitment bias and you need to fight the urge and help your client to fight the urge, too.
I hope you entered into a type of contract that compensated you for your effort in this problem resolution versus delivering the solution. If the latter, you're out of luck getting paid and you need to pay your obligations for what you outsourced. Lessons learned for the type of contract into which you need to enter for work like this.