I think you are unlikely to be at risk. You were in the open air etc and it was only one hole. That said it is not advised.
There are Asbestos fibres in the air all around us, albeit very low level, from all the Asbestos used over the years, until its use was banned.
You can get more info...