Unless the fishermen were in diving suits underneath the dropped boulders I'm not sure physical conflict is the best way to describe it.
People other, than fishermen, also go diving for recreation and work, (think research scientists and possibly others). But there are also dangers of damaging underwater equipment such as telecommunication cables, pipelines etc and also causing seabed disturbances such as mudslides if the rocks fall near the edge of a precipice.
I'm not saying the fishermen shouldn't be stopped, because I agree, they too are causing damage. What I am saying is that everyone should sit around a table and thrash out a solution thats acceptable to all.
What that solution is, I have no idea. But if they don't then even more damage will be done by both parties in the meantime.