Looking at the pic would you not end up with very narrow rooms leaving a 1m gap?
To maximise space i'd build right up to the boundary but then like people have said you run into issues such as having to have a party wall agreement, the neighbours house is no longer link detached it's a semi...