I must say you are being really decent about the whole thing. Most people I know, with young children and a nosy old neighbour poking his head over the wall every 5 minutes, would put up a new fence without even asking!
Personally I'd explain to him, given his response, you have no choise...