Yes it was a big loss and a mistake, but in the overall scheme of things, compared to say, the cost of COVID, the NHS, the welfare bill, pensions, Ukraine war etc, it's not something that bankrupted the country.
Other countries were also selling off their gold, Brown invested it in other areas...