I'd think the gravel would be okay!
Its difficult to say the cause of the damp...is there a damp proof course, is it used as presently, any ventilation etc etc
Don't bother digging down around the wall, the damp is probably because there isn't a damp proof course installed. Extra cement...