Thanks all - should be possible to get the rest of the rubble out if I take another brick or two out and drop the level of the pebbles down. It is a cavity wall.
Only other thing I’d say is that if I feel up from where I took the brick out, I can feel what seems like rough mortar that has set...