Video processing use case

Hi. I have a requirement of reading a sequence of video frames to perform a classification task. I have gone through the documentation of dali.fn.readers.video but not able to figure it out for my use case. Here is what I want to achieve:

1. I have a simple video directory structure: root/{vid1.mkv, vid2.mkv, ...}. Each video is of different duration, but have a constant fps of, say, 24. I would like to extract the center frame from a chunk of 6 consecutive frames, i.e., for 24 frames in a second ⇾ 4 chunks (each of size 6) ⇾ 1 frame from each chunk ⇾ 4 output frames. Likewise, for each second, we keep getting 4 frames.

2. It's a multi-label classification problem. If there are C classes in total, any subset of C can occur for a frame. Therefore, I have a binary vector of size C for each frame as a label. If there are n such frames in a video, the labels form a corresponding numpy array with the size n x C for that video. This array is stored in a numpy file. The label directory structure is: root/{vid1.npy, vid2.npy, ...}.

Right now, I'm doing this in 2 parts: Use ffmpeg to extract frames@24fps. Then read the selective frames into cpu before loading to gpu. Extracting frames is a one-time process, but reading frames into cpu to form a long sequence is quite slow. I was thinking to do it directly from video to frames in gpu. Any suggestions / directions would be really helpful.

Thanks!

Start with the documented dali.fn.readers.video entry point and compare its behavior with the requested root/.mkv and root/.npy layout. Check whether it can select one frame per six-frame chunk and associate per-frame multi-label arrays; the issue is complete when a supported approach or limitation is documented.