No error on GPU OOM Colabfold
Open
Nobody has claimed this yet.
bug
help wanted
- Dominant language
- HTML
- Stars
- 115
- Forks
- 75
- PR merge metrics
- No merged PRs in 30d
Description
Description of the bug
When submitting the pipeline through Seqera the platform does not evoke a 1 exit status if colabfold runs out of memory on GPU. The pipeline returns a success 0 status which prevents retries with different GPUs.
Command used and terminal output
Generated through Seqera:
nextflow run 'https://github.com/nf-core/proteinfold' -name test_run -params-file 'https://api.cloud.seqera.io/ephemeral/[id].json' -with-tower -r [tower-hash]
Relevant files
System information
- Nextflow v24.10.3
- Google Cloud
- Executor = 'google-batch'
- container Docker [COLABFOLD_BATCH:nf-core/proteinfold_colabfold:dev]
- OS Linux DeepLearning VM with CUDA preinstalled; However the CUDA on the docker image is 12.6 which solves some jax issues.
- nf-core/proteinfold v1.1.1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the attached task-6.command.out.txt and task-6.command.err.txt files, then trace how the ColabFold container reports GPU out-of-memory failures under the Google Batch executor. Reproduce the run with the provided Seqera-generated command and verify that a GPU OOM produces a nonzero status so retries can be triggered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, gcp
- Domain
- cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100