Ah, that one I know the answer to.
You have to breath it in for it to do damage. Unless your some sort of freak, you cant really inhale a sheet of anything. But the moment you drill, saw or bash it about, it releases fibers that ARE dangerous.
There is a natural percentage of fibers in the air, and we all live with them.
As to colours, I recall reading that it was an early assumption, and that in reality, there all as lethal as each other