Support for sub-sampling long videos

🚀 Feature Request

I'm trying to add support for sub-sampling long videos in StreamingDataset. I have two possible implementation methods in mind, and I'd like your feedback on which is most inline with the spirit of the codebase.

The goal is to be able to define samples as subsamples of a larger file dependency (in this case a video). I imagine that this would be sequential in nature, so samples 0-1000 are from video1, 1001-2300 are from video2, etc. Each sample would map to the frames of the video to use.

Motivation

The video datasets we're working with are very large (>10 TB) so using a frame by frame jpeg approach is impractical. On the other hand, video decoding is quite slow, so we can't be in a situation where we waste any decoding time. My hope is that we can open a single video file (1 per thread), read ALL the subsamples from it in sequence, and then move to the next video.

[Optional] Implementation

I have two potential idea for how to implement such a system, but I'd like feedback on which fits with StreamingDataset the best.

1. Each video as a sample with iterable subsamples. This approach involves defining a sample in StreamingDataset as a single video, and then adding a bit of code to the getitem and iter functions to subsample from that video until all subsamples are exhausted.

2. Each video is a single shard in StreamingDataset, and the shuffle logic is updated to pull all samples from a single shard before moving to the next shard.

Additional context

Would it be okay to call:
self.flush_shard() self._reset_cache()
manually on MDSWriter to force it to start a new shard?

Start by reading StreamingDataset’s getitem and iter paths, then inspect MDSWriter’s flush_shard() and _reset_cache() behavior. Compare the two proposed video-subsampling designs against sequential per-video decoding and shard boundaries. Done would require an agreed implementation direction and support for mapping video frames to samples without unnecessary decoding.