Just had a joiner come to look at this mess, he is supposed to be pretty good.
He said two of the holes I'd drilled in the plaster are infront of joists, the other two are infront of a beam. It's one of these which I'd drilled into which he says is fine.
While obviously not a good thing...