Do you know what the construction of the wall is?
My guess, and it is just a wild guess, is that it's (from the wall surface inwards) tile, plasterboard, stud frame with void in between, brick/stone.
To clear up the size confusion, the suggestion is that you've drilled the hole too big...