Hi, i cant see why not, My father in law just bought a house with a garage like the one you describe and one of the panels has been omitted to leave way for a door - narrow though it may be.
The only problems is see are how are you going to remove the panel, some of them have a lip or tongue...