sokrypton / sokrypton/ColabFold

illegal memory access error with GPU notebook

Open
#383 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
2.9k
Forks
747
PR merge metrics
No merged PRs in 30d

Description

Expected Behavior

Runs succeed

Current Behavior

Runs fail with an illegal memory access error

Steps to Reproduce (for bugs)

  1. connect to GPU backend on https://colab.research.google.com/github/sokrypton/ColabFold/blob/main/batch/AlphaFold2_batch.ipynb
  2. do runs with use_amber & use_templates (unsure if this matters)

ColabFold Output (for bugs)

All sequences (even short ones) give output like this:

2023-02-14 14:09:44,037 Query 35/59: 7s7icomplexColSep (length 329) 2023-02-14 14:10:16,352 Sequence 0 found templates: ['1hyr_C', '1k5n_A', '7lfj_D', '7lfm_D', '1nez_A', '3nfn_A', '7mja_A', '1je6_A', '7jyv_A', '6m2k_A', '1zs8_E', '1c16_G', '1r3h_C', '6at5_A', '7edo_A', '6eny_F', '1s7v_D', '6ddm_C', '6ddv_C'] 2023-02-14 14:11:20,929 Sequence 1 found templates: ['7km6_H', '6wzn_A', '3zhk_B', '1igt_B', '1oaq_H', '1fgv_H', '5grv_L', '7cu5_A', '6kyz_B', '5gs1_N', '7n8h_C', '5grw_B', '7orb_A', '1hez_B', '1dee_D', '6wzn_A', '2kh2_B', '4rav_C', '7mhy_O', '2kh2_B'] 2023-02-14 14:11:53,738 Sequence 2 found templates: ['6j9o_L', '7jo8_B', '4kv5_E', '4hjj_L', '6tcs_A', '1eeq_B', '6ehv_X', '5gry_G', '7bep_I', '7ekb_L', '5ujz_G', '7d0c_F', '5ujz_G', '6tcs_A', '4k07_H', '5grw_B', '3qrg_L', '5gs1_M', '5gry_G', '7vyt_C'] 2023-02-14 14:11:57,850 Padding length to 339 2023-02-14 14:13:02,945 Could not predict 7s7icomplexColSep. Not Enough GPU memory? INTERNAL: Failed to launch CUDA kernel: fusion_862 with block dimensions: 32x1x1 and grid dimensions: 5264x1x1: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with batch/AlphaFold2_batch.ipynb from the linked Colab reproduction and run the reported GPU case with use_amber and use_templates. Investigate the CUDA_ERROR_ILLEGAL_ADDRESS failure after template discovery and padding; done means the affected runs complete successfully without the illegal memory access error.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook
Domain
bioinformatics, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.