You say the clutch is ok but are you sure it is adjusted correctly? Your symptoms suggest a slightly dragging clutch. If you floor the c/pedal when stationary, engage first and slowly release pedal, at what position does it start to bite? If immediately you need to adjust the free play. If not your problem is more complicated.
Your post suggests that you change down into first while the car is still moving. This is quite brutal on the 1st/2nd synchro, so after 100k miles of this maybe you need a new one. Before doing this though, check the obvious - is the selector linkage in good condition? Check the stabiliser bar bush for excessive play. Check g/box oil and top up/replace if necessary(I know, obvious but easier than taking the box out!)Check for foreign objects around the base of the gearstick (peel rubber boot off for this) Etc...