google-deepmind / google-deepmind/alphafold

Unable to initialize backend 'gpu'

Open
#498 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
14.9k
Forks
2.9k
PR merge metrics
No merged PRs in 30d

Description

I think I managed to correctly install all the dependencies and requirements as they are described in the documentation.
When I run the command `docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi` to check if I'm able to run CUDA within Docker, I get the following output:

```
Tue Jun 7 21:34:50 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.129.06 Driver Version: 470.129.06 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro K600 Off | 00000000:03:00.0 Off | N/A |
| 25% 46C P8 N/A / N/A | 57MiB / 981MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 Quadro K600 Off | 00000000:04:00.0 Off | N/A |
| 25% 46C P8 N/A / N/A | 5MiB / 982MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
```

So, it seems Docker can at least see my GPU.
Afterwards, I built the docker image using `docker build -f docker/Dockerfile -t alphafold .` and then installed the dependencies using `pip3 install -r docker/requirements.txt`.
But, when I try to run the `run_docker.py` script I get a log message stating that no GPU could be found.
The exact command I ran is:
```
python3 ../alphafold/docker/run_docker.py \
--fasta_paths=./7mkr.fasta \
--model_preset=monomer \
--max_template_date=2020-05-14 \
--data_dir=/media/$USER/'My Passport' \
--output_dir=/home/$USER/Documentos/AlphaFold/7mkr
```

The script seems to run fine all the way to completion, but I get the following message:
```
I0607 14:44:25.218496 140038534334272 run_docker.py:255] I0607 21:44:25.217542 139683978950464 tpu_client.py:54] Starting the local TPU driver.
I0607 14:44:25.218731 140038534334272 run_docker.py:255] I0607 21:44:25.218080 139683978950464 xla_bridge.py:212] Unable to initialize backend 'tpu_driver': Not found: Unable to find driver in registry given worker: local://
I0607 14:44:25.329375 140038534334272 run_docker.py:255] I0607 21:44:25.328556 139683978950464 xla_bridge.py:212] Unable to initialize backend 'gpu': Internal: no supported devices found for platform CUDA
I0607 14:44:25.329644 140038534334272 run_docker.py:255] I0607 21:44:25.328836 139683978950464 xla_bridge.py:212] Unable to initialize backend 'tpu': Invalid argument: TpuPlatform is not available.
I0607 14:44:25.329788 140038534334272 run_docker.py:255] W0607 21:44:25.328952 139683978950464 xla_bridge.py:215] No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
```

Also, I can't find the output files generated by AlphaFold; they're not in the folder specified by `--output_dir` and there's no folder `/tmp/alphafold`, though I suppose this is a separate issue.

To check if it's a problem with the GPU drivers or with Docker, I had the idea to create a new (simple) program that uses CUDA and run it in a separate Docker image to see if it also fails to find the GPU.
But I realized I don't even know which component is sending this message; I don't know which of all the components described in the dockerfile and in the `run_docker.py` script is the one that is telling me it cannot find the GPU.
Because of this, I don't even know how to start diagnosing this problem.
I'd appreciate if somebody could point me in the right direction.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.