ColabFold process stuck in 'pending' state when using custom template and single sequence mode

Expected Behavior

The ColabFold process should complete the analysis of the input sequence using the custom template and single sequence mode.

Current Behavior

The process gets stuck in a 'pending' state and doesn't progress beyond this point for over an hour.

Steps to Reproduce (for bugs)

1. Set up the environment with the following key configurations:
   • Custom template path: --custom-template-path ./templates
   • Single sequence mode: --msa-mode single_sequence
2. Run the following bash script:

#!/bin/bash
# Define the paths
DB_PATH='/home/lily/amelie/Workspace/ColabFold_old/MsaServer/databases'
INPUT_FASTA='/home/lily/amelie/Workspace/LigandMPNN/outputs/Cx43_Xenopus-laevis/RRNYRRNY-analogues/test_custom_templates_colabfold/seqs/Cx43_Xenopu_laevis_P16863-cropped_RRNYRRNY__id_3_relaxed_rank_001_alphafold2_multimer_v3_model_1_seed_043.fa'
OUTPUT_PATH="/home/lily/amelie/Workspace/ColabFold_old/work/outputs/Cx43-Xenopu_laevis/Cx43-RRNYRRNY-analogues-3_B"
LOG_PATH="/home/lily/amelie/Workspace/ColabFold_old/work/logs"
CACHE_PATH="/home/lily/amelie/Workspace/ColabFold_old/alphafold2_cache"
CUSTOM_TEMPLATE_PATH='/home/lily/amelie/Workspace/ColabFold_old/work/outputs/templates/1'

# Advanced settings
MSA_MODE="single_sequence"
PAIR_MODE="unpaired_paired"
PAIR_STRATEGY="greedy"
USE_TEMPLATES=True
NUM_RECYCLE=10
NUM_SEEDS=3
RANDOM_SEED=42
NUM_MODELS=1
MODEL_TYPE="auto"
USE_DROPOUT=False
MAX_SEQ=6000
NUM_RELAX=1
RELAX_MAX_ITERATIONS=200
USE_GPU_RELAX=True
RANK="multimer"

# GPU settings
NUM_GPUS=4

# Run Docker container
time docker run --gpus all \
  -v "${DB_PATH}:/database" \
  -v "${INPUT_FASTA}:/input.fasta" \
  -v "${OUTPUT_PATH}:/predictions" \
  -v "${CUSTOM_TEMPLATE_PATH}:/templates" \
  -v "${LOG_PATH}:/logs" \
  -v "${CACHE_PATH}:/cache" \
  ghcr.io/sokrypton/colabfold:1.5.5-cuda11.8.0 \
  /bin/bash -c "colabfold_batch \
    --msa-mode ${MSA_MODE} \
    --templates \
    --custom-template-path ./templates \
    --pair-mode ${PAIR_MODE} \
    --pair-strategy ${PAIR_STRATEGY} \
    --num-recycle ${NUM_RECYCLE} \
    --num-seeds ${NUM_SEEDS} \
    --random-seed ${RANDOM_SEED} \
    --num-models ${NUM_MODELS} \
    --model-type ${MODEL_TYPE} \
    $([[ $USE_DROPOUT == True ]] && echo '--use-dropout') \
    --max-seq ${MAX_SEQ} \
    --num-relax ${NUM_RELAX} \
    --relax-max-iterations ${RELAX_MAX_ITERATIONS} \
    $([[ $USE_GPU_RELAX == True ]] && echo '--use-gpu-relax') \
    --rank ${RANK} \
    /input.fasta /predictions && \
  echo 'Batch processing completed'"

ColabFold Output (for bugs)

Here's the output from running the bash script:

(base) lily@il-gpu04:~/amelie/Workspace/ColabFold_old/work$ bash predict_fast4.sh
SUBMIT:   0%|          | 0/300 [elapsed: 00:00 remaining: ?]2024-10-12 23:30:00,298 Running colabfold 1.5.5
2024-10-12 23:30:01,776 Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: CUDA Interpreter
2024-10-12 23:30:01,776 Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
2024-10-12 23:30:03,392 Running on GPU
2024-10-12 23:30:03,984 Found 6 citations for tools or databases
2024-10-12 23:30:04,344 Query 1/1: Cx43_Xenopu_laevis_P16863-cropped_RRNYRRNY__id_3_relaxed_rank_001_alphafold2_multimer_v3_model_1_seed_043__id_1__T_0.15__seed_42__overall_confidence_0.1588__ligand_confidence_1.0000__seq_rec_0.1250 (length 387)
PENDING:   0%|          | 0/300 [elapsed: 10:02 remaining: ?]

Additionally, here's the content of the log.txt file:

2024-10-12 22:20:14,280 Running colabfold 1.5.5
2024-10-12 22:20:15,403 Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: CUDA Interpreter
2024-10-12 22:20:15,403 Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
2024-10-12 22:20:17,293 Running on GPU
2024-10-12 22:20:17,967 Found 6 citations for tools or databases
2024-10-12 22:20:18,644 Query 1/1: Cx43_Xenopu_laevis_P16863-cropped_RRNYRRNY__id_3_relaxed_rank_001_alphafold2_multimer_v3_model_1_seed_043__id_1__T_0.15__seed_42__overall_confidence_0.1588__ligand_confidence_1.0000__seq_rec_0.1250 (length 387)
2024-10-12 22:20:19,523 Sleeping for 5s. Reason: PENDING
2024-10-12 22:20:25,395 Sleeping for 8s. Reason: PENDING
...

Context

We are trying to run ColabFold with a custom template and in single sequence mode. The process gets stuck in a 'pending' state and doesn't progress for over an hour. We're unsure if this is related to our specific configuration, an issue with ColabFold 1.5.5, or a problem with our input or environment.

Question

How can we determine why the process is stuck in a pending state, and what steps can we take to resolve this issue?

Start with the colabfold_batch invocation in predict_fast4.sh and the log.txt lines showing the transition from SUBMIT to PENDING. Compare the custom-template and single_sequence options, the mounted paths, and the pending polling output. Done means identifying why the process remains pending and documenting a reproducible resolution or required configuration.