Sorry I wouldn't rely on a supplied seal to act as a draught excluder from the ventilated cavity, I would have made good first and then used it to finish the job.
You might aswell drill a deliberately oversized hole to aid the fitting of the flue knowing the seal will cover the gap.
Just...