You could chuck more gravel down, but if whole thing isn’t on a decent compacted sub base (suggests it’s not when you say the gravel’s being pushed down into a softer base) you’ll continue to have issues.
Proper job is to remove existing gravel, see what the base is, if it’s not solid like at...