unpackdb stalls

Expected Behavior

splitting of MSAs works multi-threaded
might be related to #189

Current Behavior

on larger final.a3m files unpackdb stalls and does not continue when executed via colabfold_search multithreaded.
manual execution single-threaded works nicely

Steps to Reproduce (for bugs)

Works not on my final.a3m

unpackdb /scratch-cbe/users/handler/alphafold_batch/2022-07-10_UAP+eIF-bait_vs_UAP+eIF-targets/MSAs.memoryTest.200/final.a3m /scratch-cbe/users/handler/alphafold_batch/2022-07-10_UAP+eIF-bait_vs_UAP+eIF-targets/MSAs.memoryTest.200 --unpack-name-mode 0 --unpack-suffix .a3m
MMseqs Version:         678c82ac44f1178bf9a3d49bfab9d7eed3f17fbc
Unpack name mode        0
Unpack suffix           .a3m
Threads                 176
Verbosity               3

[=%

works on the very same final.a3m

unpackdb /scratch-cbe/users/handler/alphafold_batch/2022-07-10_UAP+eIF-bait_vs_UAP+eIF-targets/MSAs.memoryTest.200/final.a3m /scratch-cbe/users/handler/alphafold_batch/2022-07-10_UAP+eIF-bait_vs_UAP+eIF-targets/MSAs.memoryTest.200 --unpack-name-mode 0 --unpack-suffix .a3m

MMseqs Version:         678c82ac44f1178bf9a3d49bfab9d7eed3f17fbc
Unpack name mode        0
Unpack suffix           .a3m
Threads                 1
Verbosity               3

[=================================================================] 100.00% 400 57s 152ms
Time for processing: 0h 0m 58s 195ms

ColabFold Output (for bugs)

there is no specific error message. unpackdb stalls

Context

When I execute colabfold_search on smaller input files (20) everything works nicely. On a larger set (200) I ran into this issue. The script produces some of the individual a3m files but stalls after a while.
Also, this step does not use the threads I defined upon startup of colabfold_search but uses all available threads on this node. As I did not request all of them for my SLURM job it cannot utilize that many. Do you think this might be the problem?

Your Environment

Include as many relevant details about the environment you experienced the bug in.
Mmseqs2 version 678c82ac44f1178bf9a3d49bfab9d7eed3f17fbc executed in a singularity container running ubuntu 20.04

Slurm job with 60 CPUs (120 threads) and 1000GB of memory.

Thank you,
Dominik

Start by reproducing the supplied unpackdb command on the reported final.a3m with 176 threads, then compare it with the single-threaded run and the colabfold_search path. Inspect how unpackdb is invoked from colabfold_search and how the requested thread count reaches it. Done means the larger input completes without stalling and the job does not use more threads than requested.