"mem_align could not allocate memory." in colabfold_search.sh

Hi,

After setting up and indexing the databases locally (as described on colabfold.mmseqs.com) on our HPC, I am running into trouble when trying to execute the colabfold_search.sh script. Despite running it with up to 700GB memory reserved, it returns the following output on stderr:

mem_align could not allocate memory.

and in the stdout a simple 'alignment died' message at the end:

>>> starting createdb
createdb query.fasta result//qdb 

Converting sequences
[
Time for merging to qdb_h: 0h 0m 0s 0ms
Time for merging to qdb: 0h 0m 0s 0ms
Database type: Aminoacid
Time for processing: 0h 0m 0s 50ms
>>> search
Create directory result//tmp
search result//qdb /path/to/database//uniref30_2103_db result//res result//tmp --num-iterations 3 --db-load-mode 3 -a -s 1 -e 0.1 --max-seqs 10000 

prefilter result//qdb /path/to/database//uniref30_2103_db.idx result//tmp/12655625715225010734/pref_0 --sub-mat aa:blosum62.out,nucl:nucleotide.out --seed-sub-mat aa:VTML80.out,nucl:nucleotide.out -s 1 -k 0 --k-score seq:2147483647,prof:2147483647 --alph-size aa:21,nucl:5 --max-seq-len 65535 --max-seqs 10000 --split 0 --split-mode 2 --split-memory-limit 0 -c 0 --cov-mode 0 --comp-bias-corr 1 --diag-score 1 --exact-kmer-matching 0 --mask 1 --mask-lower-case 0 --min-ungapped-score 15 --add-self-matches 0 --spaced-kmer-mode 1 --db-load-mode 3 --pca substitution:1.100,context:1.400 --pcb substitution:4.100,context:5.800 --threads 32 --compressed 0 -v 3 

Index version: 16
Generated by:  GITDIR-NOTFOUND
ScoreMatrix:  VTML80.out
Query database size: 1 type: Aminoacid
Estimated memory consumption: 97G
Target database size: 29291635 type: Aminoacid
Process prefiltering step 1 of 1

k-mer similarity threshold: 174
Starting prefiltering scores calculation (step 1 of 1)
Query db start 1 to 1
Target db start 1 to 29291635
[=================================================================] 1 0s 198ms

1.195565 k-mers per position
5430 DB matches per sequence
0 overflows
0 queries produce too many hits (truncated result)
59 sequences passed prefiltering per query sequence
59 median result list length
0 sequences with 0 size result lists
Time for merging to pref_0: 0h 0m 0s 0ms
Time for processing: 0h 0m 55s 879ms
align result//qdb /path/to/database//uniref30_2103_db.idx result//tmp/12655625715225010734/pref_0 result//tmp/12655625715225010734/aln_0 --sub-mat aa:blosum62.out,nucl:nucleotide.out -a 1 --alignment-mode 2 --alignment-output-mode 0 --wrapped-scoring 0 -e 0.1 --min-seq-id 0 --min-aln-len 0 --seq-id-mode 0 --alt-ali 0 -c 0 --cov-mode 0 --max-seq-len 65535 --comp-bias-corr 1 --max-rejected 2147483647 --max-accept 2147483647 --add-self-matches 0 --db-load-mode 3 --pca substitution:1.100,context:1.400 --pcb substitution:4.100,context:5.800 --score-bias 0 --realign 1 --realign-score-bias -0.2 --realign-max-seqs 2147483647 --corr-score-weight 0 --gap-open aa:11,nucl:5 --gap-extend aa:1,nucl:2 --zdrop 40 --threads 32 --compressed 0 -v 3 

Index version: 16
Generated by:  GITDIR-NOTFOUND
ScoreMatrix:  VTML80.out
Compute score only
Query database size: 1 type: Aminoacid
Target database size: 29291635 type: Aminoacid
Calculation of alignments
Error: Alignment died

This particular example was run for a query sequence of 496 amino acids, with 200GB of RAM reserved, running on 8 cores. The same error is produced when running with 700GB RAM reserved, or with just one core as well. Have you perhaps encountered this error before?

Thanks for all the amazing work as well!
Kind regards,
Jasper

Start with colabfold_search.sh and reproduce the reported search using query.fasta, the indexed uniref30_2103_db database, and the shown alignment command. Inspect the transition from prefiltering to align, including the reported memory estimate and the “mem_align could not allocate memory” error. The issue does not define a confirmed cause or a specific expected fix.