The asbestos survey only detailed asbestos was on the roof of the building. But drilled through some sort of insulation board which looks like it could be asbestos?
From the images posted it "appears" that the plaster has straw embedded in it??
What age is the property?
I take it it is the blue material that is of concern? the material behind the plaster??
Only absolutely sure way to find out what the material is, is to have the material you are wary of tested, you may get a good price for a test from the company that undertook your Asbestos Survey? If not google testing facilities for Asbestos.
I'd presume the material you are worried about is the top brown coloured board. This is fiberboard, occasionally this can have a thin layer of asbestos paper over the top as a vapour barrier. As another poster said only way is get it tested.