From what I've seen there's an awful lot of interpolation and putting words into mouths going on.
He has every right to ask Muslim women to remove the veil when talking to him, just as they have every right to refuse. If both sides are grown ups then no-one would see it as anything other...