Unlikely to burn house down, but could mean you get a nasty shock.
The neutral and earth are normally at the same potential, but under fault conditions, and the fault may not even be in your home, the neutral can raise well above the 50 volt deemed safe.
There are light switches which don't need a neutral, however they tend to have more of a problem with lights not switching off, and a shimmer when switched on. And also more expensive.