Modifying SAM for semantic segmentation

I know this is a pretty hot topic that has been discussed and has many different implementations, but among them I found the most simple was just to calculate loss with respect to the multiple mask outputs SAM predicts.

It appears that the authors of these papers found success with this method but could someone please explain if this is actually feasible?

To the best of my knowledge, SAM predicts masks and uses IOU prediction heads and MLP hypernetworks. My question boils down to whether it would be possible to train the network by calculating loss with respect to multiple mask outputs (e.g. 3 different masks tasked with finding different semantic classes).

A few examples of repositories that do this include https://github.com/hitachinsk/SAMed and https://github.com/ziqi-jin/finetune-anything. Thanks for the help!