Large video and loading in batches

I want to segment very large video files - and have come across https://github.com/facebookresearch/sam2/issues/264 for possible solutions.
Running with `vos_optimized=True` gives me some headache (https://github.com/facebookresearch/sam2/issues/501).

I wonder if there is an easier way to handle these things that is also more general. For example, is it possible to load the video in batches and somehow make use of the memory created for one tracked batch in the next, etc.?
What would have to be done to do that?
If I initiate only like the first 100 frames of one video (let's say it has 10,000 frames total), what are the issues here for loading and predicting in batches.