Get a knife and slice round the switch, and gently tap the switch off the wall.
Have a look, see what you got.
Then fit a new blanking plate or a new switch, for the time being.
If you use a blanking plate, you may need to terminate the feed wires in connector blocks first.
EDIT, turn ALL the power off first, and test the wiring with a proper tester. Those neon pen testers, I think this is what you used to test with, shouldn't be relied on too much - they can give false indication, so you should always double check with a proper voltage tester.
I don't want this to go into a long discussion on the pros and cons of these 'contactless' testers here, they do have their uses, and can give guidance, but double checking is important.