I experienced an internal cable short once. I didn't want to believe it because of the work required to replace it and the fact that we like to think that cables are the least likley cause of problems. However, it does seem to be your problem. I would suggest that, for diagnosis, you temporarily run a shadow cable between the switch and the lights, exactly replacing the existing run, using the same RCD, switch and lamps, etc. If the fault disappears, the cable is the cause. I think you mentioned that you intend to do something like that anyway. If you end up replacing the cable, you should consider SWA.