speed skim

I have a bathroom ceiling to plaster that has easily-removable downlights fitted. With the downlights removed would I be able to use a speed skim, or would it be problematic because of the downlight holes? I wonder if I would end up pushing plaster into the hole and find myself in a real mess? TIA