ok well I carried on knocking it down, and just as I'd hoped, the cavity stopped halfway down :-)
I carefully removed all the breeze blocks, and have carefully stacked them back up again in the cavity on my neighbours side. It's not going anywhere now.
Rather frustratingly, I couldn't get it...