GPU not recognised

Description of the bug

I am running my pipeline on a system with a single RTX 4090:

nvidia-smi 
Tue Jun 17 14:43:43 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.42.06              Driver Version: 555.42.06      CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:73:00.0 Off |                  Off |
|  0%   39C    P8             28W /  450W |       2MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

My run times out because it uses the CPU instead of the GPU (even though I have use_gpu set to true).

Jun-17 05:50:43.544 [TaskFinalizer-6] ERROR nextflow.processor.TaskProcessor - Error executing process > 'NFCORE_PROTEINFOLD:COLABFOLD:COLABFOLD_BATCH (T1024)'

Caused by:
  Process exceeded running time limit (8h)


Command executed:

  ln -r -s params/alphafold_params_*/* params/
  colabfold_batch \
      --use-gpu-relax --amber --templates \
      --num-recycle 3 \
      --data $PWD \
      --model-type alphafold2_ptm \
      T1024.a3m \
      $PWD
  for i in `find *_relaxed_rank_001*.pdb`; do cp $i `echo $i | sed "s|_relaxed_rank_|	|g" | cut -f1`"_colabfold.pdb"; done
  for i in `find *.png -maxdepth 0`; do cp $i ${i%'.png'}_mqc.png; done
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_PROTEINFOLD:COLABFOLD:COLABFOLD_BATCH":
      colabfold_batch: 1.5.2
  END_VERSIONS

Command exit status:
  -

Command output:
  2025-06-16 21:53:08,416 Running colabfold 1.5.5 (1648d2335943f9a483b6a803ebaea3e76162c788)
  2025-06-16 21:53:08,749 WARNING: no GPU detected, will be using CPU
  2025-06-16 21:53:08,823 Matplotlib created a temporary cache directory at /tmp/matplotlib-nphslddv because the default path (/home/sibbe/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
  2025-06-16 21:53:09,279 generated new fontManager
  2025-06-16 21:53:10,012 Found 9 citations for tools or databases
  2025-06-16 21:53:10,013 Query 1/1: T1024 (length 408)

Command error:
  INFO:    Converting SIF file to temporary sandbox...
  WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (350) bind mounts
  
    0%|          | 0/150 [elapsed: 00:00 remaining: ?]
  SUBMIT:   0%|          | 0/150 [elapsed: 00:00 remaining: ?]
  COMPLETE:   0%|          | 0/150 [elapsed: 00:00 remaining: ?]
  COMPLETE: 100%|██████████| 150/150 [elapsed: 00:00 remaining: 00:00]
  COMPLETE: 100%|██████████| 150/150 [elapsed: 00:01 remaining: 00:00]

Work dir:
  /home/sibbe/scratch/work/02/5e952845978d419edfceb6deb0e34d

Container:
  /home/sibbe/scratch/work/singularity/quay.io-nf-core-proteinfold_colabfold-1.1.1.img

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
Jun-17 05:50:43.552 [TaskFinalizer-6] INFO  nextflow.Session - Execution cancelled -- Finishing pending tasks before exit
Jun-17 05:50:43.568 [Actor Thread 70] DEBUG nextflow.sort.BigSort - Sort completed -- entries: 7; slices: 1; internal sort time: 0.01 s; external sort time: 0.004 s; total time: 0.014 s
Jun-17 05:50:43.576 [Actor Thread 70] DEBUG nextflow.file.FileCollector - >> temp file exists? false
Jun-17 05:50:43.577 [Actor Thread 70] DEBUG nextflow.file.FileCollector - Missed collect-file cache -- cause: java.nio.file.NoSuchFileException: /home/sibbe/scratch/work/collect-file/f21e6a94c90aca6e96ff1aacd86fd989
Jun-17 05:50:43.593 [TaskFinalizer-6] ERROR nextflow.Nextflow - Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting
Jun-17 05:50:43.594 [main] DEBUG nextflow.Session - Session await > all processes finished
Jun-17 05:50:43.595 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Jun-17 05:50:43.595 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jun-17 05:50:43.596 [Actor Thread 70] DEBUG nextflow.file.FileCollector - Saved collect-files list to: /home/sibbe/scratch/work/collect-file/f21e6a94c90aca6e96ff1aacd86fd989
Jun-17 05:50:43.601 [Actor Thread 70] DEBUG nextflow.file.FileCollector - Deleting file collector temp dir: /tmp/nxf-13813534474444119490
Jun-17 05:50:43.611 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Jun-17 05:50:43.613 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Jun-17 05:50:43.626 [main] INFO  nextflow.Nextflow - -[nf-core/proteinfold] Pipeline completed with errors-
Jun-17 05:50:43.643 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=5; failedCount=1; ignoredCount=0; cachedCount=8; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=18d 20h 37m 52s; failedDuration=2d; cachedDuration=12h 21m 6s;loadCpus=0; loadMemory=0; peakRunning=2; peakCpus=128; peakMemory=224 GB; ]
Jun-17 05:50:43.643 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jun-17 05:50:43.646 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jun-17 05:50:45.001 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jun-17 05:50:45.475 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jun-17 05:50:45.529 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin 'nf-validation@1.1.3'
Jun-17 05:50:45.530 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-validation
Jun-17 05:50:45.532 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Jun-17 05:50:45.533 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

Command used and terminal output

nextflow run nf-core/proteinfold \
      --input samples.csv \
      --outdir results \
      --mode colabfold \
      --colabfold_server local \
      --full_dbs true \
      --colabfold_model_preset "alphafold2_ptm" \
      --use_gpu true \
      -resume \
      -profile singularity \
      -c config.config

cat config.config

process { 
  withName: /NFCORE_PROTEINFOLD:PREPARE_COLABFOLD_DBS:ARIA2_.+:ARIA2/ {
    time = '12h'
  }

  withName: MMSEQS_COLABFOLDSEARCH {
    memory = '112 GB'
    cpus = 64
  }

}

Relevant files

No response

System information

No response

Start with the command invocation, config.config, and the singularity profile used to launch the container. Reproduce the run while checking the container's GPU visibility and the logged nvidia-smi warning. Done means the ColabFold task detects and uses the RTX 4090 instead of falling back to CPU.