You can, but loose cables just coming out the wall can look a bit naff IMHO, even with a brush plate. Then they get in the way in future, for decorating, new carpets, etc. Any you don't need right now are just in the way also. At least a patch lead can be unplugged.
That's neater, the wall...