I'd say if you were overtaking in single lane then you were on the wrong side of the road therefore the onus is on you to decide if it is safe to do so.
Seems harsh but sounds like because he was going slowly you passed in frustration instead of just waiting.
How did you know he was 'letting...