It is if the wealth was stolen in the first place which is likely to be the case for Baron whoever.It's not fair to attack someone for being wealthy, or having wealthy parents.
I'd rather attack someone on the basis of facts rather than "Well, you know, it's most likely that ...". Some of my ancestors were rich, but came from poor backgrounds. and who do you hold responsible for past crimes? The boat owner who sailed out to pick up slaves to sell in the Americas? The ship's crew who sailed because it was a way to make good money. Etc. Anyway, some in the Labour party (Harriet Harman etc), and a few Tories, have helped us avoid this dilemma as they are loathsome in their own right, and we need not search their family tree.