In my opinion.....
If the payment has been made for the scaffolding by the builder, you can write directly to the scaffolder, and advise them of the service charge for use of your land, at the daily rate of £50, commencing from 2 working days after delivery of the letter.
I suppose ultimately, his scaffolding has been abandoned on your property.
Sorry for your situation though, for most of us it would be a big irritation but its more serious for you.