Mice will get in no matter what you do - they can get in the tiniest of holes.
Clean up all crumbs and put down traps that KILL! Dont go all lovey dovey as wven if you take the live mouse 100m from your house to release it it will be back in your house before you are.
Put the traps down...