Extremely slow `expandaln` step in `colabfold_search`

Expected Behavior

colabfold_search executes for 10 sequences on the hours timescale

Current Behavior

colabfold_search for 10 sequences essentially hangs on expandaln (very slow progress bar, it will take 24 hours at this rate)

Note I am running it with 40 threads

Steps to Reproduce (for bugs)

I setup the databases locally without and index.

I had originally tried doing it with an index and db-load-mode 0 since I have NVMe storage. This completely hangs on the first prefiltering step and makes no progress. I thought maybe it had to do with limitations of the node I am on on my HPC cluster (104 cores, 200 GB)

I only have access to nodes with a max of 750 GB RAM, so I don't think in memory index is an option

Is this speed expected given I am indexing on the fly? If not, how can I optimize the speed without in memory index?

Thanks for your time and work.

Start by reproducing colabfold_search for 10 sequences with 40 threads and a locally installed database without an index. Compare the expandaln behavior with an index and with db-load-mode 0; done means establishing whether the runtime is expected and identifying a supported optimization path for nodes limited to 750 GB RAM.