The best thing of course is to come to an amicable agreement
This is by far and away the best solution of all.
I would go as far as you can to try and reach an agreement and remain friendly with your neighbour (even if they are obnoxious bar-stewards!)
if you enter an agreement as to who owns what? and where you plan to cut back to!! you might wish you had not said anything in the future!! as a verbal agreement is as i see it, a binding agreement/contract to what is the future agreement between you and next door neighbour for any related boundary line issues
seek advice and think long and hard about what you would like to do with your garden, and most of all, never ever give up on any boundary line, nor accept any kind of agreement in writing or verbal.
you can ask more questions on this before doing anything but if it were me? i would cut back anything on your land, but leave 12 inches of the foliage for now, and see what you think of it over the next few months.
but it is your land by the sounds of it, he can not bully you into anything you do not want to do, and if as he states it is his boundary line? then you can always ask him to show his title deeds to show and prove that it is.
otherwise do what you wish but it is always nice to chat and let them know your plans, i found that you talk! they will delay!! you plan and they again delay, and you do! and they will delay anything to simply ruin your day or week.
seek advice
that is the best advice i can give you
seek seek seek out advice
before you talk to him any more
let us know how you get on
and dont forget to seek advice
seek advice :O)