I'm trying to visualize what your describing would it be possible to post pictures ?. Is this pipe going through the wall horizontal or in the chimney vertical. If vertical it may possibly be part of an old liner or cowl thats fallen down.
The section where the fireplace has been broken away...