Stopping draughts behind dry lined walls

I live in a house build 1650 and rennovated approx 5 years ago by a builder that we purchased it from.
Due to rising fuel bills I want to sort out a problem that we have found with cold air and draughts from behind the plasterboard linings. When I asked the builder some years ago he said the air movement was needed to stop dampness and would not look at it. No NHBC as it is a renovation.

Walls are solid stone as you would expect and have all (ground and first floor) been dry lined with (mostly) foam backed (1inch) board. But there are noticable draughts from sockets, pipes and gaps around the wall ceiling joints at ground floor (ceilings are original oak beams/boards so very uneven). Even with the foam backed board, the walls are cold to touch (more so than you would expect).
In the loft there is a visible gap behind the boarding when looking down the walls that would allow cables etc to be pushed down (great) but lots of cold air too.
I have tried to seal the gap in the loft with foam/insulation but it has not been too effective. I wonder if some air is also leaking through the old stone walls due to gaps in pointing, doesn't need much for air to get in.
Any ideas? I was thinking of pouring polystyrene beads or vermiculite down the gap behind the boards ut it would probably not fill everything.

is it possible to access the peripheries of the boards and squirt in expanding foam, to help prevent the drafts?

The only way I can see to do that is drill a load of holes around the boards and squirt in some foam then make good. I thought about that but it would be big job for the whole house with holes spaced I guess at 200mm centres so the foam forms a continuous seal.