google-deepmind / google-deepmind/alphafold
Attempting to run in AWS - not detecting GPU
- Dominant language
- Python
- Stars
- 14.9k
- Forks
- 2.9k
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I am trying to setup the environment in AWS using standard DLAMI (tested with latest 20.04 / 11.8 CUDA as well as the 18.04 Deep Learning AMI GPU CUDA 11.1.1 (Ubuntu 18.04) 20230406)
Following this guide https://aws.amazon.com/blogs/machine-learning/run-alphafold-v2-0-on-amazon-ec2/ and this guide https://medium.com/proteinqure/alphafold-quickstart-on-aws-9ba20692c98e both fail.
I would appreciate any pointers if anyone managed to get this running in AWS,
In 18.04 with 11.1.1 cuda, I am getting following:
python3 /datavol/alphafold/docker/run_docker.py --fasta_paths=/datavol/input/T1024.fasta --max_template_date=2020-05-14
Traceback (most recent call last):
File "/datavol/alphafold/docker/run_docker.py", line 22, in
from absl import app
ModuleNotFoundError: No module named 'absl' even though the module shows as installed:
** pip3 show absl-py
Name: absl-py
Version: 1.4.0
Summary: Abseil Python Common Libraries, see https://github.com/abseil/abseil-py.
Home-page: https://github.com/abseil/abseil-py
Author: The Abseil Authors
Author-email:
License: Apache 2.0
Location: /home/ubuntu/.local/lib/python3.7/site-packages
Requires:
Required-by: tensorboard, tensorflow**
docker run --rm -it --gpus=all --entrypoint bash -v /usr/local/cuda-11.1/:/usr/local/cuda-11.1 -e LD_LIBRARY_PATH=/usr/local/cuda:/usr/local/cuda-11.1/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 alphafold
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: mount error: stat failed: /var/lib/docker/overlay2/2ed3c802be295259f29cb24c37402fc1adf45a2d412ef0851c8c6118b679cb54/merged/usr/local/cuda-11.1/compat/libcuda.so.455.45.01: no such file or directory: unknown.
with 20.0.4, cuda 11.8 - no issues starting the process but GPU is not getting detected:
....
I0524 19:27:05.789226 140336074209088 run_docker.py:258] I0524 19:27:05.788572 139697805058176 xla_bridge.py:353] Unable to initialize backend 'tpu_driver': NOT_FOUND: Unable to find driver in registry given worker:
I0524 19:27:07.957564 140336074209088 run_docker.py:258] I0524 19:27:07.956949 139697805058176 xla_bridge.py:353] Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: Host Interpreter CUDA
I0524 19:27:07.958004 140336074209088 run_docker.py:258] I0524 19:27:07.957579 139697805058176 xla_bridge.py:353] Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
I05
Contributor guide
Assessment
This issue has not been assessed yet.