Question: Issue with automated model ranking in mmseqs notebook?

Yesterdays protein modeling gave the following log output with regard to the ranking of the five generated models:

`2023-09-22 18:52:35,633 alphafold2_ptm_model_1_seed_000 recycle=0 pLDDT=88.1 pTM=0.799
2023-09-22 18:54:13,264 alphafold2_ptm_model_1_seed_000 recycle=1 pLDDT=87.9 pTM=0.814 tol=0.512
2023-09-22 18:55:50,591 alphafold2_ptm_model_1_seed_000 recycle=2 pLDDT=88.6 pTM=0.824 tol=0.155
2023-09-22 18:57:27,719 alphafold2_ptm_model_1_seed_000 recycle=3 pLDDT=88.5 pTM=0.824 tol=0.218
2023-09-22 18:57:27,721 alphafold2_ptm_model_1_seed_000 took 416.7s (3 recycles)
2023-09-22 18:59:05,717 alphafold2_ptm_model_2_seed_000 recycle=0 pLDDT=89.2 pTM=0.827
2023-09-22 19:00:43,084 alphafold2_ptm_model_2_seed_000 recycle=1 pLDDT=89 pTM=0.839 tol=0.563
2023-09-22 19:02:20,305 alphafold2_ptm_model_2_seed_000 recycle=2 pLDDT=89.3 pTM=0.844 tol=0.184
2023-09-22 19:03:57,420 alphafold2_ptm_model_2_seed_000 recycle=3 pLDDT=88.9 pTM=0.836 tol=0.221
2023-09-22 19:03:57,422 alphafold2_ptm_model_2_seed_000 took 389.0s (3 recycles)
2023-09-22 19:05:35,861 alphafold2_ptm_model_3_seed_000 recycle=0 pLDDT=88.6 pTM=0.832
2023-09-22 19:07:13,289 alphafold2_ptm_model_3_seed_000 recycle=1 pLDDT=89.3 pTM=0.853 tol=0.514
2023-09-22 19:08:50,580 alphafold2_ptm_model_3_seed_000 recycle=2 pLDDT=89.4 pTM=0.853 tol=0.197
2023-09-22 19:10:27,531 alphafold2_ptm_model_3_seed_000 recycle=3 pLDDT=89 pTM=0.849 tol=0.124
2023-09-22 19:10:27,533 alphafold2_ptm_model_3_seed_000 took 389.0s (3 recycles)
2023-09-22 19:12:05,644 alphafold2_ptm_model_4_seed_000 recycle=0 pLDDT=88.9 pTM=0.831
2023-09-22 19:13:42,834 alphafold2_ptm_model_4_seed_000 recycle=1 pLDDT=89.1 pTM=0.841 tol=0.595
2023-09-22 19:15:19,840 alphafold2_ptm_model_4_seed_000 recycle=2 pLDDT=89.4 pTM=0.848 tol=0.162
2023-09-22 19:16:57,128 alphafold2_ptm_model_4_seed_000 recycle=3 pLDDT=88.9 pTM=0.842 tol=0.148
2023-09-22 19:16:57,131 alphafold2_ptm_model_4_seed_000 took 388.9s (3 recycles)
2023-09-22 19:18:35,480 alphafold2_ptm_model_5_seed_000 recycle=0 pLDDT=88.8 pTM=0.848
2023-09-22 19:20:12,740 alphafold2_ptm_model_5_seed_000 recycle=1 pLDDT=88.9 pTM=0.854 tol=0.752
2023-09-22 19:21:49,979 alphafold2_ptm_model_5_seed_000 recycle=2 pLDDT=89 pTM=0.852 tol=0.344
2023-09-22 19:23:27,137 alphafold2_ptm_model_5_seed_000 recycle=3 pLDDT=88.4 pTM=0.843 tol=0.172
2023-09-22 19:23:27,139 alphafold2_ptm_model_5_seed_000 took 388.8s (3 recycles)

2023-09-22 19:23:27,833 reranking models by 'plddt' metric
2023-09-22 19:23:27,834 rank_001_alphafold2_ptm_model_3_seed_000 pLDDT=89 pTM=0.849
2023-09-22 19:23:27,835 rank_002_alphafold2_ptm_model_4_seed_000 pLDDT=88.9 pTM=0.842
2023-09-22 19:23:27,835 rank_003_alphafold2_ptm_model_2_seed_000 pLDDT=88.9 pTM=0.836
2023-09-22 19:23:27,836 rank_004_alphafold2_ptm_model_1_seed_000 pLDDT=88.5 pTM=0.824
2023-09-22 19:23:27,837 rank_005_alphafold2_ptm_model_5_seed_000 pLDDT=88.4 pTM=0.843
2023-09-22 19:23:31,615 Done`

Not sure whether I have missed anything here with the data interpretation, but I assumed the automated ranking is based on maximum pLDDT value for which each recycle state is inspected individually for each generated model - and if pLDDT is equal, then the model with the lower pTM is chosen. If this is correct, shouldn't the ranking in the above log read as follows?

2023-09-22 19:15:19,840 alphafold2_ptm_model_4_seed_000 recycle=2 pLDDT=89.4 pTM=0.848 tol=0.162
2023-09-22 19:08:50,580 alphafold2_ptm_model_3_seed_000 recycle=2 pLDDT=89.4 pTM=0.853 tol=0.197
2023-09-22 19:02:20,305 alphafold2_ptm_model_2_seed_000 recycle=2 pLDDT=89.3 pTM=0.844 tol=0.184
2023-09-22 19:21:49,979 alphafold2_ptm_model_5_seed_000 recycle=2 pLDDT=89 pTM=0.852 tol=0.344
2023-09-22 18:55:50,591 alphafold2_ptm_model_1_seed_000 recycle=2 pLDDT=88.6 pTM=0.824 tol=0.155

Start by locating the mmseqs notebook's model-reranking step and inspect how the metric named in “reranking models by 'plddt' metric” is computed from the model outputs. Compare that behavior with the per-recycle pLDDT and pTM values in the report. Done means the observed ordering is explained or corrected and verified against this example.