Actually, most shops, offices, hospitals, public buildings, railway stations, public lavatories, restaurants, pubs and even village halls now have accessible facilities that are not sex-specific.
So there is a male toilet, a female toilet, and a non specific toilet, which is typically also available for baby changing and for disabled people, whatever their sex.
In such cases, a person who, for example, is a man, and is not entitled to use the women's room, and does not want to use the mens room, can use the non-specific room.