google-deepmind / google-deepmind/alphafold

Attempting to run in AWS - not detecting GPU

Open
#765 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
14.9k
Forks
2.9k
PR merge metrics
No merged PRs in 30d

Description

Hi, I am trying to setup the environment in AWS using standard DLAMI (tested with latest 20.04 / 11.8 CUDA as well as the 18.04 Deep Learning AMI GPU CUDA 11.1.1 (Ubuntu 18.04) 20230406)

Following this guide https://aws.amazon.com/blogs/machine-learning/run-alphafold-v2-0-on-amazon-ec2/ and this guide https://medium.com/proteinqure/alphafold-quickstart-on-aws-9ba20692c98e both fail.

I would appreciate any pointers if anyone managed to get this running in AWS,

In 18.04 with 11.1.1 cuda, I am getting following:

python3 /datavol/alphafold/docker/run_docker.py --fasta_paths=/datavol/input/T1024.fasta --max_template_date=2020-05-14
Traceback (most recent call last):
File "/datavol/alphafold/docker/run_docker.py", line 22, in
from absl import app
ModuleNotFoundError: No module named 'absl' even though the module shows as installed:

** pip3 show absl-py
Name: absl-py
Version: 1.4.0
Summary: Abseil Python Common Libraries, see https://github.com/abseil/abseil-py.
Home-page: https://github.com/abseil/abseil-py
Author: The Abseil Authors
Author-email:
License: Apache 2.0
Location: /home/ubuntu/.local/lib/python3.7/site-packages
Requires:
Required-by: tensorboard, tensorflow**

docker run --rm -it --gpus=all --entrypoint bash -v /usr/local/cuda-11.1/:/usr/local/cuda-11.1 -e LD_LIBRARY_PATH=/usr/local/cuda:/usr/local/cuda-11.1/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 alphafold
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: mount error: stat failed: /var/lib/docker/overlay2/2ed3c802be295259f29cb24c37402fc1adf45a2d412ef0851c8c6118b679cb54/merged/usr/local/cuda-11.1/compat/libcuda.so.455.45.01: no such file or directory: unknown.

with 20.0.4, cuda 11.8 - no issues starting the process but GPU is not getting detected:
....
I0524 19:27:05.789226 140336074209088 run_docker.py:258] I0524 19:27:05.788572 139697805058176 xla_bridge.py:353] Unable to initialize backend 'tpu_driver': NOT_FOUND: Unable to find driver in registry given worker:
I0524 19:27:07.957564 140336074209088 run_docker.py:258] I0524 19:27:07.956949 139697805058176 xla_bridge.py:353] Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: Host Interpreter CUDA
I0524 19:27:07.958004 140336074209088 run_docker.py:258] I0524 19:27:07.957579 139697805058176 xla_bridge.py:353] Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
I05

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.