He's obviously being very unprofessional & maybe should let him know that. There may be issues to resolve but at the end of the day he can't physically touch you & if he's being volatile write letters or emails (don't text) to resolve the situation as you then have a record & don't have to...
Just to add a little more info, the neighbouring roof is asbestos corrugated, which is about 8 inches above the line of the flat roof then decreasing to about 8 inches below at the other end. The neighbouring wall is not rendered. The neighbours wall can't be touched i.e fixings.
Unfortunately in this day and age people seem to think they have some sort of "entitlement" to life, so if it means if they have to work for it, they don't want it. Far easier to sponge and put it on loans or benefits.