I'd considered that, but I suppose I was looking to see if somebody else had been there and done it!
The first baffle (a metal frame with a vermiculite brick insert) obviously restricts the pull of air and will stay, but it's the second cast iron baffle restricting it even further. I'll run it...