Twice I thought to answer and twice I deleted my reply because I could not find something useful to say. Here is attempt 3.
Your grey "filler" might have been hardwall or simply mortar. You should have persevered with sanding (thicker grade paper, electric sander, chisel, scraper, more elbow...