It definitely sounds like a limited video processing problem, my old nvr when the cameras are set to 4mp wont show the live image on the monitor just a blank screen but will record them at that resolution.
It may be just how you dvr is and nothing you can do about it, just have a mess around...