In the main the fan has two jobs, one remove smells, the other remove moisture, in both cases the air needs to get back into the room, be it a wall vent or door vent some where air needs to return into the room, and that air needs heating in winter, so we should really use heat recovery units so air going out heats air coming in, and we are sure of the path for return air, however we all try doing it on the cheap, and so we compromise, as to if the compromise is good enough depends on usage, so for my wife and I in a house with three bathrooms we really don't need a fan, non of the bathrooms get enough use to get a damp problem, and our food preference means smells are minimum.
In last house fan was turned off, there was no mould build up, and enough air movement without it needing to run, house before that really did need a fan, but did not have one, and before that it had hot air central heating and the problem was to stop house being super dry, so there is no straight answer to your question, however in the main if room is remaining damp and you have mould problem, it is usually because home is too well sealed and air can't get back into the room.