Question about video loading speed in two different pipelines

Hi @JanuszL and the team, hope you're doing well (CC'd @themattinthehatt)

We are using two different video reading pipelines in our lightning-pose repository.

1. The first scans a video and has sequence_length=64 and step=64 -- and is working very fast.

2. The second was tailored to our temporal context network, which takes in five video frames at a time and predicts just the third (middle) frame. it uses sequence_length=5, step=1 and batch_size=16.

Both are running on device="gpu". Changing the num_threads=4 argument doesn't make any difference, we do not see any meaningful cpu utilization with or without it (why? not sure, see top -i). Pipeline 1 does use 6%-8% GPU, Pipeline 2 uses approximately 0% GPU (as seen in watch -n 0.5 nvidia-smi.

I prepared a notebook and demo video for you. To reproduce

Installation

git clone https://github.com/danbider/lightning-pose.git
cd lightning-pose
python -m pip install -r requirements.txt
git checkout loss-landscapes

The reproducible notebook is here.

Question -- should CPU usage be low in general? Why is GPU usage low for pipeline 2? How could I accelerate it?

Your help is appreciated, as always