GPU VideoReader not working

🐛 Describe the bug

from torchvision.io import VideoReader
import urllib.request
import matplotlib.pyplot as plt
%matplotlib inline

urllib.request.urlretrieve("https://downloads.videezy.com/system/protected/files/000/004/210/4.mp4", "/tmp/cat.mp4")

used_timestamps = sorted(np.random.choice(np.arange(0, 8, 0.1), 10, replace=False).tolist())

video_reader_cpu = VideoReader("/tmp/cat.mp4", device="cpu", num_threads=4)


images_cpu = []
for seek in used_timestamps:
    video_reader_cpu.seek(seek)
    frame = next(video_reader_cpu)
    images_cpu.append(frame["data"].permute(1,2,0))

    
video_reader_gpu = VideoReader("/tmp/cat.mp4", device="cuda")

images_gpu = []
for seek in used_timestamps:
    video_reader_gpu.seek(seek)
    frame = next(video_reader_gpu)
    images_gpu.append(frame["data"].cpu())

for i1, i2 in zip(images_cpu, images_gpu):
    plt.figure()
    plt.subplot(121).imshow(i1)
    plt.subplot(122).imshow(i2)

When seeking to specific timestamps in the video and trying to extract the closest image frames, the cpu implementation of VideoReader works exactly as expected. However, the gpu implementation outputs progressively more corrupted versions of a single frame, with halo effects of other frames getting more prevalent in the latter frames.

Versions

Collecting environment information...
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.27

Python version: 3.8.12 (default, Oct 12 2021, 13:49:34)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.15.0-163-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.3.109
GPU models and configuration:
GPU 0: NVIDIA RTX A5000
GPU 1: NVIDIA RTX A5000
GPU 2: NVIDIA RTX A5000
GPU 3: NVIDIA RTX A5000
GPU 4: NVIDIA RTX A5000
GPU 5: NVIDIA RTX A5000
GPU 6: NVIDIA RTX A5000
GPU 7: NVIDIA RTX A5000

Nvidia driver version: 495.29.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.931
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.2
[pip3] pytorch-lightning==1.6.0rc1
[pip3] torch==1.11.0
[pip3] torchelastic==0.2.2
[pip3] torchmetrics==0.7.3
[pip3] torchtext==0.12.0
[pip3] torchvision==0.13.0a0+1db8795
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               ha36c431_9    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py38h7f8727e_0
[conda] mkl_fft                   1.3.1            py38hd3c417c_0
[conda] mkl_random                1.2.2            py38h51133e4_0
[conda] mypy                      0.931                    pypi_0    pypi
[conda] mypy-extensions           0.4.3                    pypi_0    pypi
[conda] numpy                     1.21.2           py38h20f2e39_0
[conda] numpy-base                1.21.2           py38h79a1101_0
[conda] pytorch                   1.11.0          py3.8_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-lightning         1.6.0rc1                 pypi_0    pypi
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchelastic              0.2.2                    pypi_0    pypi
[conda] torchmetrics              0.7.3                    pypi_0    pypi
[conda] torchtext                 0.12.0                     py38    pytorch
[conda] torchvision               0.13.0a0+1db8795          pypi_0    pypi