If you lose your cash someone can go on a spree until it's gone.
I don't regularly walk around with several thousand in my pocket, maybe £20 or £40. So the potential losses of someone having access to my bank account via a contactless card are much greater.