It depends on whether you mixed the sand with cement to bed the slabs? If so, I can't see it being a problem, it will still bind as with any other loose mortar. If not then as Woody said, may be best to re-lay them.
Personally as you've done all the work, I'd just keep tabs on it, when/if it...