Any idea how long the fence has been up? Historically it was polite to give the neighbour the ‘nice’ side, but more recently whoever is paying chooses to have the nice side.
Far from definitive, and not all that helpful. Sorry!
As Notch says, just get on with it. It can’t be trespass if...