Filling hole in internal wall - Not sure what material

[Apophany]

Hey,

I've just taken out some self screw plasterboard fixings in order to do some painting, and they've left quite a large hole in the wall which contains some very crumbly material. It's an internal wall, but I'm not sure if it's plasterboard, lathe and plaster, or something else I don't know about.

Either way, one of the fixings seems to have gone the whole way through the wall. I've got some advanced polyfilla (Polyfilla) but I think the hole might be too deep. I'm pretty bad with DIY since I over analyse but any recommendations on the most straight forward way to fix it and which recommended products?

Thanks!

2nd hole, not quite as deep:

 

[Harry Bloomfield]

Just fill them with filler, after removing any dust and loose material - it'll be fine.

 

[Wayners]

lath and plaster wall?
Got any wood glue? It's PVA.
Mix 50/50 Ish with water.
After a quick vacuum out the dust. Lash the PVA in hole . Give it 2 hours or more to dry. Mix up some filler. Fill, when set fill again.
Let it dry then sand

 

[Apophany]

Just fill them with filler, after removing any dust and loose material - it'll be fine.

So I went ahead with just using the filler, seemed like the quickest thing to do. Unfortunately doesn't seem to have worked as the filler never hardened properly even after 2 days, so the texture is still a bit like putty if you press on it.

I've tried applying another layer of the filler on top of that but doesn't seem to have made much difference, I can still push the filler with my thumb and it makes a dent, so seems I need something that will dry a bit harder in the quantity required for the hole

Or tips welcome for how to improve what I've done to achieve a harder finish