rebuilding the known split ids set ended up not halving its size

Describe the bug
this log line get spammed a bit:

2024-02-06T18:46:28.603Z  WARN quickwit_indexing::actors::merge_planner: Rebuilding the known split ids set ended up not halving its size. Please report. This is likely a bug, please report. known_split_ids_len_after=2017 known_split_ids_len_before=3017
2024-02-06T18:47:04.205Z  WARN quickwit_indexing::actors::merge_planner: Rebuilding the known split ids set ended up not halving its size. Please report. This is likely a bug, please report. known_split_ids_len_after=1117 known_split_ids_len_before=2117
2024-02-06T18:52:56.775Z  WARN quickwit_indexing::actors::merge_planner: Rebuilding the known split ids set ended up not halving its size. Please report. This is likely a bug, please report. known_split_ids_len_after=2881 known_split_ids_len_before=3811

there are 3 indexes, 2 empty and one with 600 to 2.5k splits (this keep going up and down), and around 350 partitions

Steps to reproduce (if applicable)
Steps to reproduce the behavior:

1. create index with ~350 partitions
2. index a few 100GB on a single indexer

Expected behavior
no warns

Configuration:
running current main (aad9b349996550bd6fec7154a9759080cf6751bd)