I think you're thinking that there's some kind of shadowy alliance of everyone that has any opinion that disagrees with whatever you think about anything.
Sadly this is the way politicians and the media are going. Seemingly anyone anti-immigration must be pro-Israel. Anyone pro-gay must, bizarrely and contradictingly, be pro-muslim.
Logic and common sense is rapidly vanishing from almost all debates. Which is why I'm intrigued by who you think "them" is - I suspect it's the "far right" cartoon characters you've been conditioned to believe in, which don't actually exist.