OK, hoping for a quick fix, this is the route I went for. It is now four days since I filled the hole and although the gripfill has now set it is still quite soft, certainly not strong enough to take the weight of the curtain pole.
Looks like I am going to have to dig it out and go the noggin...