merge pipeline gets killed when we lose the last pipeline for a given index.

(in reality it should finish it works?)

No more indexing pipeline on this index and source, killing merge pipeline. index_id=simian_chico_1434753389610517491 source_id=_ingest-source",

INFO 2024-01-16T07:09:48.674817923Z [labels."k8s-pod.statefulsetKubernetesIo.pod-name": simian-indexer-2] [resource.labels.containerName: quickwit] 2024-01-16T07:09:48.674Z INFO index-doc-batches{index_id=simian_chico_11078199147437666629 source_id=_ingest-source pipeline_uid=01HM8BQK3WQ6XTM66EHZ3YSXSZ workbench_id=01HM8HMWN33V3ZGH80HVY1FW47}:publisher{split_update=SplitsUpdate { index_id: "simian_chico_11078199147437666629", new_splits: "01HM8HN0FSV0A6HJ0Y6YWJFSGY", checkpoint_delta: Some(_ingest-source:∆(00000000000000000001:(00000000000002183048..00000000000002201061])) }}: quickwit_indexing::actors::publisher: publish-new-splits new_splits=["01HM8HN0FSV0A6HJ0Y6YWJFSGY"] checkpoint_delta=Some(_ingest-source:∆(00000000000000000001:(00000000000002183048..00000000000002201061]))
INFO 2024-01-16T07:09:49.085182304Z [labels."k8s-pod.statefulsetKubernetesIo.pod-name": simian-indexer-2] [resource.labels.containerName: quickwit] 2024-01-16T07:09:49.085Z INFO index-doc-batches{index_id=simian_chico_4208842373069537462 source_id=_ingest-source pipeline_uid=01HM8GAKYX48GNTF9BHPDAEVQS workbench_id=01HM8HNHASNT34KT1SC8AGVCD4}:uploader:upload{split=01HM8HNJ5GB1QKY3FT7MPKDSDE}:store_split: quickwit_indexing::split_store::indexing_split_store: store-split-remote-success split_size_in_megabytes=42.14941 num_docs=61317 elapsed_secs=0.3499808 throughput_mb_s=120.433495 is_mature=false
INFO 2024-01-16T07:09:49.085206794Z [labels."k8s-pod.statefulsetKubernetesIo.pod-name": simian-indexer-2] [resource.labels.containerName: quickwit] 2024-01-16T07:09:49.085Z INFO index-doc-batches{index_id=simian_chico_4208842373069537462 source_id=_ingest-source pipeline_uid=01HM8GAKYX48GNTF9BHPDAEVQS workbench_id=01HM8HNHASNT34KT1SC8AGVCD4}:uploader:upload{split=01HM8HNJ5GB1QKY3FT7MPKDSDE}:store_split: quickwit_indexing::split_store::indexing_split_store: store-in-cache
INFO 2024-01-16T07:09:49.104051912Z [labels."k8s-pod.statefulsetKubernetesIo.pod-name": simian-indexer-2] [resource.labels.containerName: quickwit] 2024-01-16T07:09:49.103Z INFO index-doc-batches{index_id=simian_chico_4208842373069537462 source_id=_ingest-source pipeline_uid=01HM8GAKYX48GNTF9BHPDAEVQS workbench_id=01HM8HNHASNT34KT1SC8AGVCD4}:publisher{split_update=SplitsUpdate { index_id: "simian_chico_4208842373069537462", new_splits: "01HM8HNJ5GB1QKY3FT7MPKDSDE", checkpoint_delta: Some(_ingest-source:∆(00000000000000000001:(00000000000007214124..00000000000007230332] 00000000000000000013:(00000000000001249945..00000000000001259509] 00000000000000000016:(..00000000000000035544])) }}: quickwit_indexing::actors::publisher: publish-new-splits new_splits=["01HM8HNJ5GB1QKY3FT7MPKDSDE"] checkpoint_delta=Some(_ingest-source:∆(00000000000000000001:(00000000000007214124..00000000000007230332] 00000000000000000013:(00000000000001249945..00000000000001259509] 00000000000000000016:(..00000000000000035544]))
INFO 2024-01-16T07:09:52.399143239Z [labels."k8s-pod.statefulsetKubernetesIo.pod-name": simian-indexer-2] [resource.labels.containerName: quickwit] 2024-01-16T07:09:52.398Z INFO quickwit_indexing::actors::indexing_service: No more indexing pipeline on this index and source, killing merge pipeline. index_id=simian_chico_1434753389610517491 source_id=_ingest-source
{
"textPayload": "2024-01-16T07:09:52.398Z INFO quickwit_indexing::actors::indexing_service: No more indexing pipeline on this index and source, killing merge pipeline. index_id=simian_chico_1434753389610517491 source_id=_ingest-source",
"insertId": "xkwhiln2ikiktf6w",
"resource": {
"type": "k8s_container",
"labels": {
"container_name": "quickwit",
"location": "us-east1-d",
"project_id": "quickwit-prod",
"namespace_name": "simian",
"cluster_name": "quickwit-prod-gke",
"pod_name": "simian-indexer-2"
}
},
"timestamp": "2024-01-16T07:09:52.399143239Z",
"severity": "INFO",
"labels": {
"compute.googleapis.com/resource_name": "gke-quickwit-prod-gk-quickwit-prod-gk-c4ded138-1ega",
"k8s-pod/controller-revision-hash": "simian-indexer-b4d75c649",
"k8s-pod/app_kubernetes_io/component": "indexer",
"k8s-pod/app_kubernetes_io/instance": "release-name",
"k8s-pod/statefulset_kubernetes_io/pod-name": "simian-indexer-2",
"k8s-pod/app_kubernetes_io/name": "quickwit"
},
"logName": "projects/quickwit-prod/logs/stdout",
"receiveTimestamp": "2024-01-16T07:09:57.228070200Z"
}

Start at the quickwit_indexing::actors::indexing_service log entry and trace how indexing and merge pipelines are stopped when no pipeline remains for an index and source. Reproduce the reported sequence if possible; done means establishing whether the merge is incorrectly killed and adding a verified fix or regression coverage.