I must admit I've not tried it myself, but I'm intrigued, and slightly worried by what you state.I just tried drilling some with a 5mm masonry bit - took about 3 seconds per hole.
May I ask what type of drill you were using (percussion or rotary hammer) and at what speed?