Want to use matt over silk

[Dagz]

So, after preparation I decided for the silk and painted two coats. The walls are not perfect so I want to switch to matt. Browsing internet I found out it is a good idea to either sand the paint or to use a primer. I lean towards the primer as there are some areas that are difficult to sand (like a cm gap between door frame and wall).
But also reading some people used matt over gloss without any problem. Is it wort to risk it with Valspar v500 paint to go with matt directly on the silk?
There is a fast acting primer that is a bit costly (Zinsser). Is there a cheaper primer that dries fast so I can decorate the room in these two days I have off?
Thanks.

 

[Ghummy]

I had the very same issue couple of weeks ago. Old walls painted with really dark coloured silk. We were going matt white

Very large open plan victorian house with tall ceilings which would have taken me all year to sand. Ended up doing 2 coats of zinsser 123 followed by 2 coats of dulux trade brilliant white matt. End result was the dogs danglies