fine-tune the model so that it could segment blood vessels

I found that there were not many medical images in SAM's training data set, such as retinal blood vessel images, so I wanted to fine-tune the model so that it could segment blood vessels. I used the DRIVE data set, input the result of coarse segmentation using Unet network into the prompt_encoder, and then fine-tune the predicted value generated by SAM and calculate the ground loss. I don't know whether this is feasible or what are the loopholes in my thinking? I would appreciate your reply