Hi Andy,
No, a fence post shouldn't be loose. As it's concrete, it won't be an issue of rotting. It could be that the hole wasn't wide/deep/square enough. That would mean there isn't enough concrete to secure the post properly.
If you were hit by any storms in the past few months this...
I don't think it would look odd at all. We have timber fencing on top of a low level brick wall. Not really the same thing but the timber and brick contrast works well.
Also if you have a bit of a ridge it might make the water run away from the wall.
I would normally recommend gravel of some kind. Then you can choose a colour that would suit. But if you have a problems with things like that running away cobbles might look good.