Well, I know what I'd tell someone who wanted to charge me 'travelling time'! Does anyone here charge travelling time? What if your car breaks down and you have to walk or rely on public transport - will you charge more? If you're self employed, your travelling time is your problem - you should have included that in your quote if you feel the client should pay that. Looks like you've just started out - weren't you the one asking for someone for you to watch put a fence up the other week? Just learn by your 'mistake'. You're just starting out, aren't you? You might have just got a regular customer - be grateful, do a good job and that will lead on to further work.
On the other hand, if that's your way of pricing a job, why not charge them for a squeeze of toothpaste, breakfast, lunch, laundry money for your work clothes, tax, mot and car insurance plus a bit of servicing and wear and tear on your car while while you are at it?