Finetune video predictor on custom data

I was able to finetune SAM on custom image segmentation dataset which trained the mask decoder and got very high IoU. Then I used the same model weights to test video predictor but didn't see improvement. I assume it has to do with memory encoder. How can I finetune the whole model? Didn't want to reinvent the wheel if someone has done it.

The issue names no file, test, or entry point. Start with the repository's example notebooks and available checkpoint or training guidance, then determine whether whole-model fine-tuning for the video predictor is supported; done means a documented or implemented path that addresses custom-data video prediction.

Written by the indexing model from the issue text.