It depends where you feel the wall; the heat from the room passes out to the world through the wall. How quickly it does so is a function of how good an insulator the wall is. If it's a poor insulator it conducts heat away quickly and will forever feel cool, if you're on the inside.. If you've been standing out in the freezing cold for a while and your fingers have gone numb, then putting them on the uninsulated wall will leave you with the impression that the wall is warm
There's a temperature gradient across most things, even the room air itself. The fire is arguably hundreds or thousands of degrees, yet you don't burst into flames. The temperature of the air surrounding the fire is hot, the air particles slam into each other with great energy, and the energy radiates away from the source like moving through a crowd. Imagine skateboarding through a busy street, relying on the momentum from a single push at the beginning, every time you bump into someone you jostle them along in some random direction, but you lose some of your own energy til eventually you're just moving around randomly at the rate people can slam into you from various directions. Good heat conductors like metal are like everyone being in neat rows and every slam moves you along in a progressive direction with a good amount of pace. Insulators don't do a good job of transmitting the jostling so a large commotion can build up on one side of the city and you're not affected on the other. While the side of the insulator touching the heat source will feel warm, the other side feels cool
If your wall is composite insulator and conductor, the final temperature of the conductor part depends on where the insulator is. If you put kingspan on the inside, the temperature drop across the kingspan means the room at 20, not much of the 20 gets through the wall, it then reaches a material that quickly conducts it away to the world at 0 and while the middle of the brick might be at 1 or 2 degrees the outer will be at 0 and this is how the gradient is shaped, 20 down to 1 across the kingspan and 1 down to 0 across the brick
Put the insulation on the outside and the gradient will still form the same, the fire will spend a long time heating the brick and the brick will eventually come up to nearly the temp of the room so the gradient is like, 20 to 19 across the brick, and 19 to 0 across the kingspan. If there is no kingspan the gradient across the brick includes some of the room air too (actually, it would with the kingspan but I've simplified that out) so the gradient across the brick is 1 down to 0, because it's that good at conducting heat, so it's the few inches of air next to the wall that have to accept the gradient of 20 to 1 forming across them. This causes them to cool and starts a convection draught, and it causes the surface of the wall to feel cold. This is also why there are often concerns with insulating old walls internally.. Cooling the wall to the point where condensation will form on the inner surface of the brick can lead to damp problems. Conversely insulating the outside and leaving the heating going will warm the brick well above dew point and the temperature gradient across the kingspan includes the temp where condensation forms. As no moist air can get into the kingspan and be there at the point the temp hits the magic number, no condensation/damp occurs
It's important to note that even an insulated wall can feel cold if you've only just turned a heat source on. All that I've discussed to now regards touching door handles and insulated door panels assumes a steady state of the heating having been on for x time and the temperature gradients have had chance to establish and become steady in their heat flows
