colabfold_search fails on Prefilter Stage in Colabfold Singularity Image

Hello,
I am running Colabfold via singularity on a HPC server. I am able to run predictions (that uses the MSA server). I want run predictions for a lot of sequences, so I need precompute their MSAs so I don't use the MSA server. I keep getting Error: Prefilter died whenever I try to run colabfold_search (via the singularity image) to precompute my the MSA.

Below are the Steps I followed

• I pulled and installed ColabFold Sigularity image using instructions here. I am using this version: colabfold:1.5.5-cuda12.2.2

• I then downloaded the db required locally. I used this command MMSEQS_NO_INDEX=1 ./setup_databases.sh /path/to/my/db | tee -a ./db_dwnld4.log. In my db folder, I have all the *_READY files (ie. COLABDB_READY colabfold_envdb_202108_db.GPU_READY DOWNLOADS_READY PDB100_READY PDB_MMCIF_READY PDB_READY uniref30_2302_db.GPU_READY UNIREF30_READY). Based on this, I am implying the db download was successful.

• I then went on to run my colab_search command via the singularity image using the following command:
  MMSEQS_IGNORE_INDEX=1 singularity run -B /scratch/.../cache_dir:/cache -B $(pwd):/work -B /scratch/.../storage:/storage colabfold_1.5.5-cuda12.2.2.sif colabfold_search /work/complex_seqs_processed_small.fasta /cache/colabfold_db4 /storage/precomputed_msa_200_5

Below is the output when I ran colab_search this way. It seem to be failing on the prefilter stage. I would appreciate it if anyone can guide me on what to do to fix this problem.

/storage/precomputed_msa_200_5/qdb exists and will be overwritten
createdb /storage/precomputed_msa_200_5/query.fas /storage/precomputed_msa_200_5/qdb --shuffle 0 

MMseqs Version:       	15.6f452
Database type         	0
Shuffle input database	false
Createdb mode         	0
Write lookup file     	1
Offset of numeric ids 	0
Compressed            	0
Verbosity             	3

Converting sequences
[
Time for merging to qdb_h: 0h 0m 0s 4ms
Time for merging to qdb: 0h 0m 0s 4ms
Database type: Aminoacid
Time for processing: 0h 0m 0s 41ms
search /storage/precomputed_msa_200_5/qdb /cache/colabfold_db4/uniref30_2302_db /storage/precomputed_msa_200_5/res /storage/precomputed_msa_200_5/tmp --threads 64 --num-iterations 3 --db-load-mode 0 -a -e 0.1 --max-seqs 10000 --k-score 'seq:96,prof:80' 

MMseqs Version:                        	15.6f452
Substitution matrix                    	aa:blosum62.out,nucl:nucleotide.out
Add backtrace                          	true
Alignment mode                         	2
Alignment mode                         	0
Allow wrapped scoring                  	false
E-value threshold                      	0.1
Seq. id. threshold                     	0
Min alignment length                   	0
Seq. id. mode                          	0
Alternative alignments                 	0
Coverage threshold                     	0
Coverage mode                          	0
Max sequence length                    	65535
Compositional bias                     	1
Compositional bias                     	1
Max reject                             	2147483647
Max accept                             	2147483647
Include identical seq. id.             	false
Preload mode                           	0
Pseudo count a                         	substitution:1.100,context:1.400
Pseudo count b                         	substitution:4.100,context:5.800
Score bias                             	0
Realign hits                           	false
Realign score bias                     	-0.2
Realign max seqs                       	2147483647
Correlation score weight               	0
Gap open cost                          	aa:11,nucl:5
Gap extension cost                     	aa:1,nucl:2
Zdrop                                  	40
Threads                                	64
Compressed                             	0
Verbosity                              	3
Seed substitution matrix               	aa:VTML80.out,nucl:nucleotide.out
Sensitivity                            	5.7
k-mer length                           	0
Target search mode                     	0
k-score                                	seq:2147483647,prof:2147483647
Alphabet size                          	aa:21,nucl:5
Max results per query                  	10000
Split database                         	0
Split mode                             	2
Split memory limit                     	0
Diagonal scoring                       	true
Exact k-mer matching                   	0
Mask residues                          	1
Mask residues probability              	0.9
Mask lower case residues               	0
Minimum diagonal score                 	15
Selected taxa                          	
Spaced k-mers                          	1
Spaced k-mer pattern                   	
Local temporary path                   	
Rescore mode                           	0
Remove hits by seq. id. and coverage   	false
Sort results                           	0
Mask profile                           	1
Profile E-value threshold              	0.1
Global sequence weighting              	false
Allow deletions                        	false
Filter MSA                             	1
Use filter only at N seqs              	0
Maximum seq. id. threshold             	0.9
Minimum seq. id.                       	0.0
Minimum score per column               	-20
Minimum coverage                       	0
Select N most diverse seqs             	1000
Pseudo count mode                      	0
Min codons in orf                      	30
Max codons in length                   	32734
Max orf gaps                           	2147483647
Contig start mode                      	2
Contig end mode                        	2
Orf start mode                         	1
Forward frames                         	1,2,3
Reverse frames                         	1,2,3
Translation table                      	1
Translate orf                          	0
Use all table starts                   	false
Offset of numeric ids                  	0
Create lookup                          	0
Add orf stop                           	false
Overlap between sequences              	0
Sequence split mode                    	1
Header split mode                      	0
Chain overlapping alignments           	0
Merge query                            	1
Search type                            	0
Search iterations                      	3
Start sensitivity                      	4
Search steps                           	1
Prefilter mode                         	0
Exhaustive search mode                 	false
Filter results during exhaustive search	0
Strand selection                       	1
LCA search mode                        	false
Disk space limit                       	0
MPI runner                             	
Force restart with latest tmp          	false
Remove temporary files                 	false

prefilter /storage/precomputed_msa_200_5/qdb /cache/colabfold_db4/uniref30_2302_db /storage/precomputed_msa_200_5/tmp/11992105744714455148/pref_0 --sub-mat 'aa:blosum62.out,nucl:nucleotide.out' --seed-sub-mat 'aa:VTML80.out,nucl:nucleotide.out' -s 5.7 -k 0 --target-search-mode 0 --k-score seq:2147483647,prof:2147483647 --alph-size aa:21,nucl:5 --max-seq-len 65535 --max-seqs 10000 --split 0 --split-mode 2 --split-memory-limit 0 -c 0 --cov-mode 0 --comp-bias-corr 1 --comp-bias-corr-scale 1 --diag-score 1 --exact-kmer-matching 0 --mask 1 --mask-prob 0.9 --mask-lower-case 0 --min-ungapped-score 15 --add-self-matches 0 --spaced-kmer-mode 1 --db-load-mode 0 --pca substitution:1.100,context:1.400 --pcb substitution:4.100,context:5.800 --threads 64 --compressed 0 -v 3 

Query database size: 1 type: Aminoacid
Estimated memory consumption: 177G
Target database size: 36293491 type: Aminoacid
Index table k-mer threshold: 122 at k-mer size 7 
Index table: counting k-mers
[=================================================================] 36.29M 12s 499ms
Index table: Masked residues: 0
No k-mer could be extracted for the database /cache/colabfold_db4/uniref30_2302_db.
Maybe the sequences length is less than 14 residues.
Error: Prefilter died
Traceback (most recent call last):
  File "/usr/local/envs/colabfold/bin/colabfold_search", line 10, in <module>
    sys.exit(main())
  File "/usr/local/envs/colabfold/lib/python3.9/site-packages/colabfold/mmseqs/search.py", line 319, in main
    mmseqs_search_monomer(
  File "/usr/local/envs/colabfold/lib/python3.9/site-packages/colabfold/mmseqs/search.py", line 91, in mmseqs_search_monomer
    run_mmseqs(mmseqs, ["search", base.joinpath("qdb"), dbbase.joinpath(uniref_db), base.joinpath("res"), base.joinpath("tmp"), "--threads", str(threads)] + search_param)
  File "/usr/local/envs/colabfold/lib/python3.9/site-packages/colabfold/mmseqs/search.py", line 25, in run_mmseqs
    subprocess.check_call([mmseqs] + params)
  File "/usr/local/envs/colabfold/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '[PosixPath('mmseqs'), 'search', PosixPath('/storage/precomputed_msa_200_5/qdb'), PosixPath('/cache/colabfold_db4/uniref30_2302_db'), PosixPath('/storage/precomputed_msa_200_5/res'), PosixPath('/storage/precomputed_msa_200_5/tmp'), '--threads', '64', '--num-iterations', '3', '--db-load-mode', '0', '-a', '-e', '0.1', '--max-seqs', '10000', '--k-score', "'seq:96,prof:80'"]' returned non-zero exit status 1.

This is my test input fasta file:

>sp|P01308|INS_HUMAN Insulin OS=Homo sapiens OX=9606 GN=INS PE=1 SV=1
MALWMRLLPLLALLALWGPDPAAAFVNQHLCGSHLVEALYLVCGERGFFYTPKTRREAED
LQVGQVELGGGPGAGSLQPLALEGSLQKRGIVEQCCTSICSLYQLENYCN

Start with the colabfold_search entry point and the traceback location in colabfold/mmseqs/search.py, then inspect the setup_databases.sh workflow and the MMseqs prefilter output for the mounted uniref30_2302_db. Reproduce the failure with the supplied insulin FASTA and Singularity command. Done means colabfold_search completes the prefilter stage and produces the requested precomputed MSA output.