The celotex is fine, it's there to prevent damp transfer and to insulate.
If you can remove the concrete, I'm assuming it's bridging the cavity on top of the celotex. Otherwise you might be better drylining the wall with a membrane, battens and plasterboard. A bit drastic but apart from losing...