Could you perhaps have some insulated panels made up, to fit into some of those windows, for use during winter? Likewise any roof lights. It will not help in summer though unless you were to leave the roof light panels in place.
The panels could even be combined with curtains/blinds - they could disguise the fact that the windows are blocked off.
Putting panels over half of the windows would likely make a noticeable difference, without cutting out too much light.
...and @twixx , I love what you have done to the garden!