Your builder was telling you the truth, you need to get below any sort of topsoil. A gravity retaining wall needs a solid base, so a concrete footing is as good as any to ensure local settlement will not cause a failure. Whichever system you are using the manufacturer will have guidelines to...