Does the OP mean that the chimney in question, or rather one side of the chimney is formed by the party wall?
If that is not the case, the only other way would be for the OP's property to have a chimney breast extending into his property, which was nothing to do with his property.