Yes it absolutely does matter if someone is using your wireless, for a start if they're downloading illegal content (music, movies etc), you're responsible by law for the consequences. In addition, you may unknowingly be sharing content with these rogue users from your own personal computer, it is not unknown for some amateur IT organisations sharing the root of the C: drive to guest users to ease remote support, in such a context the next door neighbour could take a copy of your Outlook mail store (PST) or your personal/confidential Word/Excel documents!!
Also, once they're on the same network as you (connected to your wireless), they can use very simple tools to scan the content you post to the internet, including capturing login details for sites. They could even configure the router to throttle your access speed to the internet so that they can enjoy full speed!!
There are a few simple ways to secure your wireless connection/network:
- Use a minimum of WPA encryption (do NOT use WEP)
- Use a strong encryption key that isn't guessable!
- Lock your router down with MAC filtering, this will prevent unknown computers connecting even if they know your Wireless key
- Disable UPNP, only devices YOU connect/configure should define access.
- Don't bother hiding your SSID, if you do the above there's no point/need.
- Disable WPS, its for noobs...
- Change the default password for your router admin panel.
And another suggestion that I personally do but isn't strictly required:
- Plug your router into a timer plug socket, switch off at 1am, back on at 8am (adjust to your bedtimes lol!), not only does this keep your router alive and avoid connection issues with your provider, it prevents hacking attempts whilst you sleep, and also saves a tiny bit of power!!
With the above and some common sense, you'll be fine!