It doesn't matter how free-draining it is, if it has any organic matter in (dark coloured stuff), then it's alive and moving. E.g. there are microscopic bits of plant and poo that are decomposing and shrinking, water and air voids, worms are tunneling through it etc.
You can only safely build on soil that doesn't have biology in it, i.e. anything lighter coloured. You normally dig down around a metre to find this. It's no firmer than topsoil, the point is that it's dead so not moving anywhere. Also it's undisturbed so hasn't been fluffed up over the years. Black or dark soil is alive and moving.
You should only normally build flexible stuff on top of topsoil, e.g. paving slabs, decking, fences. If you put something rigid on top of something that's moving then it will probably crack.
A dwarf wall is a reasonable gamble, if you're talking about probably trebling its height or weight or more then adding a heavier roof too then that's a risk.
Builders usually rip out and start again when replacing a conservatory with a room. I doubt they do this just for fun, more likely it's from bitter experience and knowledge.
