Thanks again Stem.
I have dug down a bit further, to find a layer of rough brick lying on the bare clay. The bricks are damp, but not sopping, and the hardcore that I dug out was generally fairly dry.
I think what I'm going to do is dry out the hardcore, then bung the bigger bits back in...