google-deepmind / google-deepmind/alphafold

Enabling Unified Memory on Compute Cluster

Open
#696 3 comments 0 reactions 0 assignees View on GitHub
out of memory
Dominant language
Python
Stars
14.9k
Forks
2.9k
PR merge metrics
No merged PRs in 30d

Description

dear all,

Some of my AF2 jobs with larger sequences are failing, for what I can only assume is memory issues. I have discovered it is possible to set up unified memory by adding the two flags below to my SLURM submission script, however the issue persists. I have been working with our IT department but we cannot resolve it. Is there any advice on what I might be doing wrong in my submission script, or other things I can check? thank you.

flags to add for unified memory:
```
export TF_FORCE_UNIFIED_MEMORY=1
export XLA_PYTHON_CLIENT_MEM_FRACTION="4.0"
```

my script (note I have removed actual paths from the script):
```
#!/bin/bash
#
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=200GB
#
#SBATCH --time=36:00:00
#SBATCH --no-requeue
#
#SBATCH --partition=gpu
#SBATCH --gres=gpu:4

MY_PROTEIN_PATH=(my_fasta_location.fasta)

echo $HOSTNAME

module load alphafold/2.2.4c

export OPENMM_CUDA_COMPILER=$(which nvcc)
export TF_FORCE_UNIFIED_MEMORY=1
export XLA_PYTHON_CLIENT_MEM_FRACTION="4.0"

python3 run_alphafold.py \
--model_preset=multimer \
--fasta_paths=$MY_PROTEIN_PATH \
--output_dir=$(dirname $MY_PROTEIN_PATH) \
--data_dir= \
--mgnify_database_path= \
--template_mmcif_dir= \
--max_template_date=2020-05-14 \
--obsolete_pdbs_path= \
--use_gpu_relax=true \
--bfd_database_path= \
--uniclust30_database_path= \
--uniref90_database_path= \
--pdb_seqres_database_path= \
--uniprot_database_path=

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.