I am leaning towards taking the rest out, it seems simpler, plus i will be doing most of the brick carrying no need for the gym :p
I should state that the chimney was not keyed into the party wall, instead there were perhaps 2 small metal rods hanging out, it seemed the chimney was stacked up...