RuntimeError: cuda video backend is not available.

🐛 Describe the bug

When trying to set the videoreader backend to cuda (torchvision.set_video_backend('cuda')) I get the error below:
RuntimeError: cuda video backend is not available.
I followed the instructions to use the videoreader on cuda. I.e. I installed pytorch nightly and build torchvision from source. My DockerFile is given below:

FROM nvidia/cuda:11.8.0-devel-ubuntu22.04

RUN apt-get update
RUN apt-get install -y python3-pip
# RUN pip3 install --upgrade pip3

RUN apt-get update
RUN yes | apt install nvidia-cuda-toolkit

RUN pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu118

RUN git clone https://github.com/pytorch/vision.git
WORKDIR "/vision"
RUN python3 setup.py develop

RUN pip3 install ffmpeg-python
RUN pip3 install av --upgrade

As far as I can see the environment has been installed with the expected versions etc. Is this a bug or am I doing something wrong?

Versions

PyTorch version: 2.2.0.dev20231213+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.10.0-25-amd64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
Nvidia driver version: 535.146.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] pytorch-triton==2.1.0+bcad9dabe1
[pip3] torch==2.2.0.dev20231213+cu118
[pip3] torchaudio==2.2.0.dev20231213+cu118
[pip3] torchvision==0.18.0.dev20231213+cu118
[conda] Could not collect

Start by reproducing the failure from the provided Dockerfile, then inspect the torchvision.set_video_backend('cuda') entry point and the source build invoked by setup.py develop. Confirm whether the CUDA video backend is built and available in this environment; done means the documented setup enables the backend or clearly identifies the missing requirement.