Don't forget that you'd only be applying pva onto the stone that's having plaster applied to it, so a very small area effectively...shouldn't be an issue.
It would be a bit of a gamble hacking off as you're right, you don't know what's underneath, but probably worth it I reckon. Any problems...