Unfortunately he does have a leg to stand on, however wobbly. You are voiding a contract, in the case of legal challenge the burden is on you to show why. And if you sue him for his mistakes, the burden is again on you to make the case, which is a seperate case.
From your description it...