Relative risk is not the issue here, though
If I get to work without incident and feel fine, then I can immediately assess the damage done.
If I breathe in asbestos without incident and feel fine,, I have no way of telling what damage is done.
There is a known danger to asbestos, it does not increase or decrease depending on what other riskesare in your life.
But to look at your comparision - Travelling to work carries a risk, but we take reasonable precautions ie tyres are sound, car is serviced, we wear a seat belt etc. We can't eliminate risk fully , but we take as many precations as we can, being aware of other users and the road conditions etc. We eliminate known risks such as driving on bald tyres and bad brakes or doing 120 on a corner
So with asbestos we should do the same, eliminate risk where ever we can.