If you prefer speed over scenery and harder terrain, go on the road.
Mountain biking will never be fast, in fact, on my regular round of 20 miles I only average 14/15mph, that's a mix of gravel, paths in woodland and some hard trails.
But the natural environment, absence of cars and other bikers make it worthwhile.
When I go with friends I don't enjoy it as much, because I have to go at their pace and mind them to avoid falling.
Don't worry if you're not as fit as you would like; start riding and within a month you'll see major progress, trust me.
If someone starts riding with you and have same level of fitness you'll enjoy it more.
Don't go with someone much faster or much slower than you.