How to speed up `colabfold_search` command on local search?

Expected Behavior

Having MSA results from running MMseqs2 locally in minutes, like from https://a3m.mmseqs.com/, not hours

Current Behavior

Running colabfold_search example.fasta /path/to/db_folder results_example takes about two hours for 280aa sequence on 64GB RAM, 32-core AMD Ryzen CPU and 2TB SSD.

In README.md you can found comment that this process should be CPU intensive:

This needs a lot of CPU
colabfold_search input_sequences.fasta /path/to/db_folder search_results

But in my case it is only I/O intensive. mmseqs uses intensively only one CPU core and I can see (in top and iotop) that mmseqs process in mostly doing I/O operations. iotop is showing about 200M/s.

Also my RAM is almost not used, most of the time real usage is below 2GB. I would like to use CPU and RAM to accelerate search speed.

This is how it looks like:

As you can see nor CPU or RAM are used intensively.

Steps to Reproduce

pip install "colabfold[alphafold] @ git+https://github.com/sokrypton/ColabFold"
./setup_databases.sh /path/to/db_folder
# download MMseqs2, use snippet found in https://github.com/sokrypton/ColabFold/blob/main/MsaServer/setup-and-start-local.sh
MMSEQS_COMMIT=edb8223d1ea07385ffe63d4f103af0eb12b2058e
wget https://mmseqs.com/archive/${MMSEQS_COMMIT}/mmseqs-linux-avx2.tar.gz

Context

I want to know how can I accelerate MSA search. How ColabFold team is able to provide search results in about 1 min 20 sec per sequence?

Let's assume that I can take from Amazon AWS any server necessary for having result in less than 2 minutes. What are the magic parameters? Or can I take cluster of ordinary computers and split db search across them, in such way that every computer is searching only in part of the DB and results are merged at the end?

I know that in my current machine I can replace SSD with NVMe disk, but I will get approx 2-3 times acceleration (and would see results in 1hour), still far too slow.

Or can I accelerate by searching many sequences at once? For example waiting for 2 hours would be acceptable for me, if after such time I could get 100 results at once.

@martin-steinegger is saying in https://github.com/sokrypton/ColabFold/issues/116#issuecomment-988580594 that:

Online searches: Our Colabfold server has ~760GB RAM and keeps full database and index in memory.

How this was made? After removing *.tar.gz and *.tsv files, results files from mmseqs tsv2exprofiledb ... and mmseqs createindex ... on uniref30_2103_db and colabfold_envdb_202108_db have about 1,3 TB (here is listing of my directory).

@martin-steinegger have you somehow compressed db produced by mmseqs, put everything in RAM disk and this allowed for fast colabfold_search run?

Is msa.sh, that can be found in results from https://a3m.mmseqs.com/, exact script which was launched on server to perform search?

Thank you for your work

Start with the README.md guidance and reproduce the shown colabfold_search command using setup_databases.sh and the local setup script at MsaServer/setup-and-start-local.sh. Compare the local database preparation and search behavior with the referenced msa.sh workflow; done means establishing documented, reproducible guidance for accelerating or batching local searches.