precondition failed for splits

Hi,
We have 4 indexes in quickwit, all indexes are in GCS, quickwit version 0.8.2.
Recently started issue with the biggest index fluentbit-logs-2024-09-12 when splits older 1h is automatically removed. Other indexes are ok. The issue is not gone after the index recreation.

The affected index has way more Staged splits than other indexes.

select count(*), split_state, index_uid from splits  group by index_uid, split_state order by index_uid;
 count |    split_state    |                          index_uid
-------+-------------------+--------------------------------------------------------------

  1941 | Staged            | fluentbit-logs-2024-09-12:01J7MNHZMJSJJ80B2JN40CE12J
   485 | Published         | fluentbit-logs-2024-09-12:01J7MNHZMJSJJ80B2JN40CE12J
  9851 | MarkedForDeletion | fluentbit-logs-2024-09-12:01J7MNHZMJSJJ80B2JN40CE12J
  8349 | MarkedForDeletion | fluentbit-logs-connect-2024-02-05:01HSA1Z6D9N74GPZ0DX9B3RVYV
   902 | Published         | fluentbit-logs-connect-2024-02-05:01HSA1Z6D9N74GPZ0DX9B3RVYV
    16 | Staged            | fluentbit-logs-connect-2024-02-05:01HSA1Z6D9N74GPZ0DX9B3RVYV
  9156 | MarkedForDeletion | fluentbit-logs-sk-2024-01-30:01HNDX8JAKT5A6A06BFVZF8TC2
  1020 | Published         | fluentbit-logs-sk-2024-01-30:01HNDX8JAKT5A6A06BFVZF8TC2
    15 | Staged            | fluentbit-logs-sk-2024-01-30:01HNDX8JAKT5A6A06BFVZF8TC2

In the indexer logs the only errors:

2024-09-16T07:13:19Z 2024-09-16T07:13:19.603Z DEBUG merge{merge_split_id=01J7WTQKDHE1DSZJCC8HZZBM6V split_ids=["01J7WTQ9BFBHCHRXT9MXZ265DA", "01J7WTKJSX9DZ5GACD8GGK3E5T", "01J7WTFVCS7S8NBTFRCN0ZNWAB", "01J7WTCEM5PXZ6BCCN8CP9YG2B", "01J7WT8X7M3YNXWTYB5SVQM6X9", "01J7WT5NG5PNZ571V6PGBMH5TZ", "01J7WT2D80WANG4V0EN4FVZHKF", "01J7WSZ07FDD1DB8SE1M8ZX2ZT", "01J7WSVJXY8G25A2Y99V580GNC", "01J7WSQS4WYMB8V98SJSA89BP4"] typ=Merge}:uploader:upload{split=01J7WTQ
KDHE1DSZJCC8HZZBM6V}:store_split: quickwit_indexing::split_store::indexing_split_store: store-split-remote-success split_size_in_megabytes=2336.5374 num_docs=5074840 elapsed_secs=34.93994 throughput_mb_s=66.87296 is_mature=false
2024-09-16T07:13:19Z 2024-09-16T07:13:19.603Z DEBUG merge{merge_split_id=01J7WTQKDHE1DSZJCC8HZZBM6V split_ids=["01J7WTQ9BFBHCHRXT9MXZ265DA", "01J7WTKJSX9DZ5GACD8GGK3E5T", "01J7WTFVCS7S8NBTFRCN0ZNWAB", "01J7WTCEM5PXZ6BCCN8CP9YG2B", "01J7WT8X7M3YNXWTYB5SVQM6X9", "01J7WT5NG5PNZ571V6PGBMH5TZ", "01J7WT2D80WANG4V0EN4FVZHKF", "01J7WSZ07FDD1DB8SE1M8ZX2ZT", "01J7WSVJXY8G25A2Y99V580GNC", "01J7WSQS4WYMB8V98SJSA89BP4"] typ=Merge}:uploader:upload{split=01J7WTQ
KDHE1DSZJCC8HZZBM6V}:store_split: quickwit_indexing::split_store::indexing_split_store: store-in-cache
2024-09-16T07:13:19Z 2024-09-16T07:13:19.604Z DEBUG merge{merge_split_id=01J7WTQKDHE1DSZJCC8HZZBM6V split_ids=["01J7WTQ9BFBHCHRXT9MXZ265DA", "01J7WTKJSX9DZ5GACD8GGK3E5T", "01J7WTFVCS7S8NBTFRCN0ZNWAB", "01J7WTCEM5PXZ6BCCN8CP9YG2B", "01J7WT8X7M3YNXWTYB5SVQM6X9", "01J7WT5NG5PNZ571V6PGBMH5TZ", "01J7WT2D80WANG4V0EN4FVZHKF", "01J7WSZ07FDD1DB8SE1M8ZX2ZT", "01J7WSVJXY8G25A2Y99V580GNC", "01J7WSQS4WYMB8V98SJSA89BP4"] typ=Merge}:uploader: quickwit_actors::ac
tor_context: from=MergeUploader-ancient-MtKx send=MergePublisher-cold-XqZ7 msg=SplitsUpdate { index_id: "fluentbit-logs-2024-09-12", new_splits: "01J7WTQKDHE1DSZJCC8HZZBM6V", checkpoint_delta: None }
2024-09-16T07:13:19Z 2024-09-16T07:13:19.635Z ERROR quickwit_actors::spawn_builder: actor-failure cause=failed to publish splits
2024-09-16T07:13:19Z
2024-09-16T07:13:19Z Caused by:
2024-09-16T07:13:19Z precondition failed for splits `01J7WT2D80WANG4V0EN4FVZHKF, 01J7WSQS4WYMB8V98SJSA89BP4, 01J7WSVJXY8G25A2Y99V580GNC, 01J7WSZ07FDD1DB8SE1M8ZX2ZT`: splits are not marked for deletion exit_status=Failure(failed to publish splits
2024-09-16T07:13:19Z
2024-09-16T07:13:19Z Caused by:
2024-09-16T07:13:19Z precondition failed for splits `01J7WT2D80WANG4V0EN4FVZHKF, 01J7WSQS4WYMB8V98SJSA89BP4, 01J7WSVJXY8G25A2Y99V580GNC, 01J7WSZ07FDD1DB8SE1M8ZX2ZT`: splits are not marked for deletion)
2024-09-16T07:13:19Z 2024-09-16T07:13:19.635Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=MergePublisher-cold-XqZ7 exit_status=failure(cause=failed to publish splits
2024-09-16T07:13:19Z
2024-09-16T07:13:19Z Caused by:
2024-09-16T07:13:19Z precondition failed for splits `01J7WT2D80WANG4V0EN4FVZHKF, 01J7WSQS4WYMB8V98SJSA89BP4, 01J7WSVJXY8G25A2Y99V580GNC, 01J7WSZ07FDD1DB8SE1M8ZX2ZT`: splits are not marked for deletion)
2024-09-16T07:13:19Z 2024-09-16T07:13:19.635Z ERROR quickwit_actors::actor_context: exit activating-kill-switch actor=MergePublisher-cold-XqZ7 exit_status=Failure(failed to publish splits
2024-09-16T07:13:19Z
2024-09-16T07:13:19Z Caused by:
2024-09-16T07:13:19Z precondition failed for splits `01J7WT2D80WANG4V0EN4FVZHKF, 01J7WSQS4WYMB8V98SJSA89BP4, 01J7WSVJXY8G25A2Y99V580GNC, 01J7WSZ07FDD1DB8SE1M8ZX2ZT`: splits are not marked for deletion)
2024-09-16T07:13:19Z 2024-09-16T07:13:19.635Z DEBUG quickwit_common::kill_switch: kill-switch-activated
2024-09-16T07:13:19Z 2024-09-16T07:13:19.813Z DEBUG quickwit_indexing::actors::merge_pipeline: Merge pipeline running. pipeline_id=IndexingPipelineId { node_id: "quickwit-indexer-27", index_uid: IndexUid { index_id: "otel-logs-v0_7", incarnation_id: Ulid(2063959136266216495463690505829921250) }, source_id: "_ingest-api-source", pipeline_uid: Pipeline(01J7WN5H0HPQKHF24X7WR6ZCWG) } generation=1 healthy_actors=["MergePlanner-spring-SN2A", "MergeSplitDownloader-throbbing-dUH8", "MergeExecutor-dawn-pAvm", "MergePackager-twilight-VTk7", "MergeUploader-black-JdUK", "MergePublisher-old-iQPj"] failed_or_unhealthy_actors=[] success_actors=[]
2024-09-16T07:13:19Z 2024-09-16T07:13:19.813Z DEBUG quickwit_indexing::actors::merge_pipeline: Merge pipeline running. pipeline_id=IndexingPipelineId { node_id: "quickwit-indexer-27", index_uid: IndexUid { index_id: "query-log-2024-06-17", incarnation_id: Ulid(2077716558178011118253688979804328215) }, source_id: "_ingest-api-source", pipeline_uid: Pipeline(01J7WN5H0HD2FE5DTTAE559ZNY) } generation=1 healthy_actors=["MergePlanner-broken-d4ur", "MergeSplitDownloader-black-YjDS", "MergeExecutor-nameless-ti3k", "MergePackager-ancient-RdhO", "MergeUploader-sparkling-RbOn", "MergePublisher-falling-k4Ls"] failed_or_unhealthy_actors=[] success_actors=[]

index config

 - version: 0.8
    index_id: fluentbit-logs-2024-09-12
    doc_mapping:
      mode: dynamic
      field_mappings:
      - name: timestamp
        type: datetime
        input_formats:
        - iso8601
        output_format: unix_timestamp_millis
        precision: milliseconds
        fast: true
      - name: message
        type: text
        tokenizer: default
        record: position
        stored: true
      - name: log
        type: text
        tokenizer: default
        record: position
        stored: true

      timestamp_field: timestamp
    indexing_settings:
      commit_timeout_secs: 10
    search_settings:
      default_search_fields:
      - message
      - log
    retention:
      period: 5 days
      schedule: hourly