On docker after some time: No compatible CUDA device is available

The Environment:
================

Running on Docker 20.10.12-3 on RHEL 7.9 with 4x NVIDIA Corporation GV100GL [Tesla V100 PCIe 32GB] cards.
CUDA version on this host is 11.4.

I also get this version reported, when doing a
```
docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
```
Seems fine to me?

The Problem:
============

So I am running alphafold (version from beginning of march 2022) like this:

```
python3 /software/alphafold2/alphafold/docker/run_docker.py \
--fasta_paths=/software/db/alphafold2/example_input/T1050.fasta \
--max_template_date=2020-05-14 --data_dir=/software/db/alphafold2/alldata
```
This is using python3 from a venv installed like this:

```
python3 -m venv $envPath
source $envPath/bin/activate
pip3 install --upgrade pip
pip3 install -r /software/alphafold2/alphafold/docker/requirements.txt
```

After quite some hours this is comming up exactly a 100 times:

```
I0316 18:50:27.090867 140103983331136 run_docker.py:247] I0316 17:50:27.090230 139823954499392 amber_minimize.py:408] Minimizing protein, attempt 100 of 100.
I0316 18:50:29.051809 140103983331136 run_docker.py:247] I0316 17:50:29.049898 139823954499392 amber_minimize.py:69] Restraining 6213 / 12189 particles.
I0316 18:50:29.170636 140103983331136 run_docker.py:247] I0316 17:50:29.170062 139823954499392 amber_minimize.py:417] No compatible CUDA device is available
```

... ultimately leading to this:

```
I0316 18:50:29.207758 140103983331136 run_docker.py:247] Traceback (most recent call last):
I0316 18:50:29.207883 140103983331136 run_docker.py:247] File "/app/alphafold/run_alphafold.py", line 445, in I0316 18:50:29.207957 140103983331136 run_docker.py:247] app.run(main)
I0316 18:50:29.208023 140103983331136 run_docker.py:247] File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 312, in run I0316 18:50:29.208087 140103983331136 run_docker.py:247] _run_main(main, args)
I0316 18:50:29.208148 140103983331136 run_docker.py:247] File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 258, in _run_main
I0316 18:50:29.208209 140103983331136 run_docker.py:247] sys.exit(main(argv))
I0316 18:50:29.208268 140103983331136 run_docker.py:247] File "/app/alphafold/run_alphafold.py", line 429, in main
I0316 18:50:29.208327 140103983331136 run_docker.py:247] is_prokaryote=is_prokaryote)
I0316 18:50:29.208385 140103983331136 run_docker.py:247] File "/app/alphafold/run_alphafold.py", line 250, in predict_structure
I0316 18:50:29.208454 140103983331136 run_docker.py:247] relaxed_pdb_str, _, _ = amber_relaxer.process(prot=unrelaxed_protein)
I0316 18:50:29.208513 140103983331136 run_docker.py:247] File "/app/alphafold/alphafold/relax/relax.py", line 66, in process
I0316 18:50:29.208570 140103983331136 run_docker.py:247] use_gpu=self._use_gpu)
I0316 18:50:29.208626 140103983331136 run_docker.py:247] File "/app/alphafold/alphafold/relax/amber_minimize.py", line 483, in run_pipeline I0316 18:50:29.208684 140103983331136 run_docker.py:247] use_gpu=use_gpu)
I0316 18:50:29.208741 140103983331136 run_docker.py:247] File "/app/alphafold/alphafold/relax/amber_minimize.py", line 419, in _run_one_iteration
I0316 18:50:29.208799 140103983331136 run_docker.py:247] raise ValueError(f"Minimization failed after {max_attempts} attempts.")
I0316 18:50:29.208857 140103983331136 run_docker.py:247] ValueError: Minimization failed after 100 attempts.
```

The Questions:
==============

* What is Alphafold (docker) considering a "compatible CUDA device"?
* As I think I have such one - what can I do to make it acknowledged by Alphafold running in docker?