sokrypton / sokrypton/ColabFold

RosettaFold2 Colab crashing at end of prediction run

Open
#603 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
2.9k
Forks
747
PR merge metrics
No merged PRs in 30d

Description

At the end of a RosettaFold2 Colab dimer prediction run (with default variables) that has run through 6 recycles––where I tried both V100 GPU & then T4 TPU settings (both with High Ram option)––the program crashes with the following error message below.

The RuntimeError: "Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat2 in method wrapper_CUDA_bmm)**

jobname: XXX
lengths: [555, 555]
getting unpaired MSA
COMPLETE: 100%|██████████| 150/150 [elapsed: 00:00 remaining: 00:00]
N=2103 L=1110
recycle 0 plddt 0.563 pae 25.219 rmsd 37.300
recycle 1 plddt 0.571 pae 24.891 rmsd 11.995
recycle 2 plddt 0.567 pae 24.844 rmsd 5.358
recycle 3 plddt 0.567 pae 24.938 rmsd 2.560
recycle 4 plddt 0.567 pae 24.922 rmsd 2.661
recycle 5 plddt 0.561 pae 24.969 rmsd 1.578
recycle 6 plddt 0.562 pae 24.969 rmsd 2.129

**RuntimeError Traceback (most recent call last)
in <cell line: 87>()
90 np.random.seed(seed)
91 npz = f"{jobname}/rf2_seed{seed}_00.npz"
---> 92 pred.predict(inputs=[f"{jobname}/msa.a3m"],
93 out_prefix=f"{jobname}/rf2_seed{seed}",
94 symm=symm,

2 frames
/usr/local/lib/python3.10/dist-packages/torch/functional.py in einsum(*args)
378 # the path for contracting 0 or 1 time(s) is already optimized
379 # or the user has disabled using opt_einsum
--> 380 return _VF.einsum(equation, operands) # type: ignore[attr-defined]
381
382 path = None

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat2 in method wrapper_CUDA_bmm)**

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Colab prediction cell at the pred.predict call shown in the traceback, then inspect the torch.einsum failure for the CPU and cuda:0 operands. Reproduce the default RosettaFold2 dimer run through six recycles using the reported GPU settings. Done means the prediction completes without the device-mismatch RuntimeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, pytorch
Domain
bioinformatics, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.