Panic happens on indexer that indexes data and in the same time the index is deleted

Description

Setup:

• one indexer is indexing some data on a given index
• the index is delete via a REST API call

A panic then happens here:

thread 'tokio-runtime-worker' panicked at 'assertion failed: split_path.is_dir()', quickwit-indexing/src/split_store/local_split_store.rs:410:9

Furthermore, a split can finally be uploaded to S3 after the deletion of the index and won't be deleted.
In the logs shown below, the split 01H7W54FF69F8QM750A8EAD2QF is not deleted (I checked it was still on S3).

Logs

2023-08-15T09:17:04.078Z  INFO quickwit_indexing::actors::indexer: send-to-index-serializer commit_trigger=Timeout split_ids=01H7W7P61TXDZ7BJ0317RJB478 num_docs=482388
2023-08-15T09:17:08.348Z  INFO merge{merge_split_id=01H7W54FF69F8QM750A8EAD2QF split_ids=["01H7W48X2QS9P24GFQJJWGNRYG", "01H7W472CDW8CEXWSJZWS92DMH", "01H7W457QX8M45A5EDRSBNBDAZ", "01H7W43CP97BA5JKGFHRH8E198", "01H7W41HJ1EHKCZXW6G3KM9RJW", "01H7W3ZPJ9EV21M4FQ0MDDMHEJ"] typ=Merge}:merge_executor: quickwit_indexing::actors::merge_executor: merge-operation-success merged_num_docs=2219197 elapsed_secs=422.20132 operation_type=Merge
2023-08-15T09:17:08.445Z  INFO merge{merge_split_id=01H7W54FF69F8QM750A8EAD2QF split_ids=["01H7W48X2QS9P24GFQJJWGNRYG", "01H7W472CDW8CEXWSJZWS92DMH", "01H7W457QX8M45A5EDRSBNBDAZ", "01H7W43CP97BA5JKGFHRH8E198", "01H7W41HJ1EHKCZXW6G3KM9RJW", "01H7W3ZPJ9EV21M4FQ0MDDMHEJ"] typ=Merge}:packager: quickwit_indexing::actors::packager: start-packaging-splits split_ids=["01H7W54FF69F8QM750A8EAD2QF"]
2023-08-15T09:17:08.445Z  INFO merge{merge_split_id=01H7W6JPW338CGB5YNQQSRSKKF split_ids=["01H7W63D15SFRQJP5HNJ3FTK21", "01H7W5Z1HDM2BM9JS6YSWVQMKS", "01H7W5W1W77C5A2G68K71BRS6T", "01H7W5T743M3A6E2FDH04HAS9N", "01H7W5QFPVGDE8X773V8YVMBWF", "01H7W5NFTC49QK3ZYC65TFDFJ0"] typ=Merge}:merge_split_downloader: quickwit_indexing::actors::merge_split_downloader: download-merge-splits dir=/quickwit/qwdata/indexing/gh-archive%01H7TXRP8NEYNS5QA00C7M0THZ%gh-archive-source%0%jpFuUz/merge%TxrITt
2023-08-15T09:17:08.445Z  INFO merge{merge_split_id=01H7W54FF69F8QM750A8EAD2QF split_ids=["01H7W48X2QS9P24GFQJJWGNRYG", "01H7W472CDW8CEXWSJZWS92DMH", "01H7W457QX8M45A5EDRSBNBDAZ", "01H7W43CP97BA5JKGFHRH8E198", "01H7W41HJ1EHKCZXW6G3KM9RJW", "01H7W3ZPJ9EV21M4FQ0MDDMHEJ"] typ=Merge}:packager: quickwit_indexing::actors::packager: create-packaged-split split_id="01H7W54FF69F8QM750A8EAD2QF"
2023-08-15T09:17:08.447Z  INFO index-doc-batches{index_id=gh-archive source_id=gh-archive-source pipeline_ord=0 workbench_id=01H7W7DQRF44TC6FHETSYHQ7FK}:uploader: quickwit_indexing::actors::uploader: start-stage-and-store-splits split_ids=["01H7W7DQRGWKHT8X34J26PNT4Q"]
2023-08-15T09:17:08.489Z  INFO index-doc-batches{index_id=gh-archive source_id=gh-archive-source pipeline_ord=0 workbench_id=01H7W7JG8KKGDZWPP1FHYJGN55}:packager: quickwit_indexing::actors::packager: start-packaging-splits split_ids=["01H7W7JG9NN3HABRRGEEJ8D2CJ"]
2023-08-15T09:17:08.489Z  INFO index-doc-batches{index_id=gh-archive source_id=gh-archive-source pipeline_ord=0 workbench_id=01H7W7JG8KKGDZWPP1FHYJGN55}:packager: quickwit_indexing::actors::packager: create-packaged-split split_id="01H7W7JG9NN3HABRRGEEJ8D2CJ"
2023-08-15T09:17:08.848Z ERROR quickwit_actors::spawn_builder: actor-failure cause=Failed to publish splits.

Caused by:
    Splits `["01H7W6RTK429JY8Q3GYV0VG6T4"]` are not staged. exit_status=Failure(Failed to publish splits.

Caused by:
    Splits `["01H7W6RTK429JY8Q3GYV0VG6T4"]` are not staged.)
2023-08-15T09:17:08.848Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=Publisher-old-YmKM exit_status=Failure(cause=Failed to publish splits.

Caused by:
    Splits `["01H7W6RTK429JY8Q3GYV0VG6T4"]` are not staged.)
2023-08-15T09:17:08.848Z ERROR quickwit_actors::actor_context: exit activating-kill-switch actor=Publisher-old-YmKM exit_status=Failure(Failed to publish splits.

Caused by:
    Splits `["01H7W6RTK429JY8Q3GYV0VG6T4"]` are not staged.)
2023-08-15T09:17:08.848Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=quickwit_indexing::actors::sequencer::Sequencer<quickwit_indexing::actors::publisher::Publisher>-morning-wYTv exit_status=Killed
2023-08-15T09:17:08.848Z  WARN index-doc-batches{index_id=gh-archive source_id=gh-archive-source pipeline_ord=0 workbench_id=01H7W7G16PYYH667CJ99E0YVCP}:uploader: quickwit_indexing::actors::uploader: Kill switch was activated. Cancelling upload. split_ids=["01H7W7G16RQWE9PY52K79NBBWP"]
2023-08-15T09:17:08.874Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=IndexUploader-late-4phl exit_status=Killed
2023-08-15T09:17:09.206Z ERROR quickwit_actors::spawn_builder: actor-failure cause=Failed to open file for read: 'IoError { io_error: Custom { kind: Other, error: "Directory kill switch was activated." }, filepath: "0b105505b1b84bd1bf1c52b09f5b31ff.fieldnorm" }'

Caused by:
    IoError: 'Custom { kind: Other, error: "Directory kill switch was activated." }' happened while opening the following file for Read: 0b105505b1b84bd1bf1c52b09f5b31ff.fieldnorm. exit_status=Failure(Failed to open file for read: 'IoError { io_error: Custom { kind: Other, error: "Directory kill switch was activated." }, filepath: "0b105505b1b84bd1bf1c52b09f5b31ff.fieldnorm" }'

Caused by:
    IoError: 'Custom { kind: Other, error: "Directory kill switch was activated." }' happened while opening the following file for Read: 0b105505b1b84bd1bf1c52b09f5b31ff.fieldnorm.)
2023-08-15T09:17:09.206Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=quickwit_indexing::actors::index_serializer::IndexSerializer-icy-haaf exit_status=Failure(cause=Failed to open file for read: 'IoError { io_error: Custom { kind: Other, error: "Directory kill switch was activated." }, filepath: "0b105505b1b84bd1bf1c52b09f5b31ff.fieldnorm" }'

Caused by:
    IoError: 'Custom { kind: Other, error: "Directory kill switch was activated." }' happened while opening the following file for Read: 0b105505b1b84bd1bf1c52b09f5b31ff.fieldnorm.)
2023-08-15T09:17:09.206Z ERROR quickwit_actors::actor_context: exit activating-kill-switch actor=quickwit_indexing::actors::index_serializer::IndexSerializer-icy-haaf exit_status=Failure(Failed to open file for read: 'IoError { io_error: Custom { kind: Other, error: "Directory kill switch was activated." }, filepath: "0b105505b1b84bd1bf1c52b09f5b31ff.fieldnorm" }'

Caused by:
    IoError: 'Custom { kind: Other, error: "Directory kill switch was activated." }' happened while opening the following file for Read: 0b105505b1b84bd1bf1c52b09f5b31ff.fieldnorm.)
2023-08-15T09:17:09.213Z ERROR quickwit_actors::spawn_builder: actor-failure cause=Failed to open file for read: 'IoError { io_error: Custom { kind: Other, error: "Directory kill switch was activated." }, filepath: ".managed.json" }'

Caused by:
    IoError: 'Custom { kind: Other, error: "Directory kill switch was activated." }' happened while opening the following file for Read: .managed.json. exit_status=Failure(Failed to open file for read: 'IoError { io_error: Custom { kind: Other, error: "Directory kill switch was activated." }, filepath: ".managed.json" }'

Caused by:
    IoError: 'Custom { kind: Other, error: "Directory kill switch was activated." }' happened while opening the following file for Read: .managed.json.)
2023-08-15T09:17:09.213Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=Indexer-floral-5d2W exit_status=Failure(cause=Failed to open file for read: 'IoError { io_error: Custom { kind: Other, error: "Directory kill switch was activated." }, filepath: ".managed.json" }'

Caused by:
    IoError: 'Custom { kind: Other, error: "Directory kill switch was activated." }' happened while opening the following file for Read: .managed.json.)
2023-08-15T09:17:09.213Z ERROR quickwit_actors::actor_context: exit activating-kill-switch actor=Indexer-floral-5d2W exit_status=Failure(Failed to open file for read: 'IoError { io_error: Custom { kind: Other, error: "Directory kill switch was activated." }, filepath: ".managed.json" }'

Caused by:
    IoError: 'Custom { kind: Other, error: "Directory kill switch was activated." }' happened while opening the following file for Read: .managed.json.)
2023-08-15T09:17:09.291Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=SourceActor-young-oXrI exit_status=Killed
2023-08-15T09:17:09.292Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=quickwit_indexing::actors::doc_processor::DocProcessor-cool-6Sjb exit_status=Killed
2023-08-15T09:17:09.314Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=Packager-proud-lFg3 exit_status=Downstream actor exited.
2023-08-15T09:17:09.314Z ERROR quickwit_actors::actor_context: exit activating-kill-switch actor=Packager-proud-lFg3 exit_status=DownstreamClosed
2023-08-15T09:17:09.429Z  INFO merge{merge_split_id=01H7W54FF69F8QM750A8EAD2QF split_ids=["01H7W48X2QS9P24GFQJJWGNRYG", "01H7W472CDW8CEXWSJZWS92DMH", "01H7W457QX8M45A5EDRSBNBDAZ", "01H7W43CP97BA5JKGFHRH8E198", "01H7W41HJ1EHKCZXW6G3KM9RJW", "01H7W3ZPJ9EV21M4FQ0MDDMHEJ"] typ=Merge}:uploader: quickwit_indexing::actors::uploader: start-stage-and-store-splits split_ids=["01H7W54FF69F8QM750A8EAD2QF"]
2023-08-15T09:17:11.365Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=Publisher-twilight-kcmp exit_status=Killed
2023-08-15T09:17:11.365Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=IndexUploader-fragrant-ZUrn exit_status=Killed
2023-08-15T09:17:11.365Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=quickwit_indexing::actors::sequencer::Sequencer<quickwit_indexing::actors::publisher::Publisher>-quiet-CAI8 exit_status=Killed
2023-08-15T09:17:11.366Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=Indexer-divine-yML1 exit_status=Killed
2023-08-15T09:17:11.366Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=IndexingPipeline-rough-iBH4 exit_status=Killed
2023-08-15T09:17:11.365Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=quickwit_indexing::actors::doc_processor::DocProcessor-morning-Fh3F exit_status=Killed
2023-08-15T09:17:11.366Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=IndexingPipeline-snowy-x8nt exit_status=Killed
2023-08-15T09:17:11.366Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=quickwit_indexing::actors::index_serializer::IndexSerializer-still-fOap exit_status=Killed
2023-08-15T09:17:11.366Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=Packager-old-Rx4N exit_status=Killed
2023-08-15T09:17:11.385Z  INFO quickwit_indexing::actors::indexing_service: queue-delete-success queue_id=gh-archive
2023-08-15T09:17:12.173Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=SourceActor-wild-5ycI exit_status=Killed
2023-08-15T09:17:36.828Z  INFO quickwit_indexing::actors::indexing_service: No more indexing pipeline on this index and source, killing merge pipeline. index_id=gh-archive source_id=gh-archive-source
2023-08-15T09:17:36.828Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=MergePipeline-damp-v5qZ exit_status=Killed
2023-08-15T09:17:36.828Z  INFO quickwit_indexing::actors::indexing_service: No more indexing pipeline on this index and source, killing merge pipeline. index_id=gh-archive source_id=_ingest-api-source
2023-08-15T09:17:36.828Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=MergePipeline-green-UvYJ exit_status=Killed
2023-08-15T09:17:37.110Z ERROR quickwit_actors::spawn_builder: actor-failure cause=An IO error occurred: 'Directory kill switch was activated.' exit_status=Failure(An IO error occurred: 'Directory kill switch was activated.')
2023-08-15T09:17:37.110Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=MergeExecutor-nameless-9kXo exit_status=Failure(cause=An IO error occurred: 'Directory kill switch was activated.')
2023-08-15T09:17:37.110Z ERROR quickwit_actors::actor_context: exit activating-kill-switch actor=MergeExecutor-nameless-9kXo exit_status=Failure(An IO error occurred: 'Directory kill switch was activated.')
2023-08-15T09:17:37.226Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=MergePackager-floral-3Tpe exit_status=Killed
2023-08-15T09:17:37.226Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=MergeSplitDownloader-crimson-YuMr exit_status=Killed
2023-08-15T09:17:37.227Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=MergeUploader-billowing-RLqp exit_status=Killed
2023-08-15T09:17:37.925Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=MergePlanner-dry-o3VB exit_status=Killed
2023-08-15T09:17:37.925Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=MergeSplitDownloader-patient-55Xp exit_status=Killed
2023-08-15T09:17:37.957Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=MergeExecutor-dry-Jiwo exit_status=Killed
2023-08-15T09:17:37.957Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=MergePackager-delicate-dor7 exit_status=Killed
2023-08-15T09:17:37.957Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=MergeUploader-snowy-AfMx exit_status=Killed
2023-08-15T09:17:37.957Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=MergePublisher-shy-hWK2 exit_status=Killed
2023-08-15T09:17:38.447Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=MergePlanner-still-eyTc exit_status=Killed
2023-08-15T09:17:49.766Z  INFO index-doc-batches{index_id=gh-archive source_id=gh-archive-source pipeline_ord=0 workbench_id=01H7W7DQRF44TC6FHETSYHQ7FK}:uploader:upload{split=01H7W7DQRGWKHT8X34J26PNT4Q}:store_split: quickwit_indexing::split_store::indexing_split_store: store-split-remote-success split_size_in_megabytes=542.16327 num_docs=382343 elapsed_secs=40.93391 throughput_mb_s=13.244844 is_mature=false
2023-08-15T09:17:49.766Z  INFO index-doc-batches{index_id=gh-archive source_id=gh-archive-source pipeline_ord=0 workbench_id=01H7W7DQRF44TC6FHETSYHQ7FK}:uploader:upload{split=01H7W7DQRGWKHT8X34J26PNT4Q}:store_split: quickwit_indexing::split_store::indexing_split_store: store-in-cache
2023-08-15T09:19:29.340Z  INFO merge{merge_split_id=01H7W54FF69F8QM750A8EAD2QF split_ids=["01H7W48X2QS9P24GFQJJWGNRYG", "01H7W472CDW8CEXWSJZWS92DMH", "01H7W457QX8M45A5EDRSBNBDAZ", "01H7W43CP97BA5JKGFHRH8E198", "01H7W41HJ1EHKCZXW6G3KM9RJW", "01H7W3ZPJ9EV21M4FQ0MDDMHEJ"] typ=Merge}:uploader:upload{split=01H7W54FF69F8QM750A8EAD2QF}:store_split: quickwit_indexing::split_store::indexing_split_store: store-split-remote-success split_size_in_megabytes=3469.235 num_docs=2219197 elapsed_secs=139.75017 throughput_mb_s=24.82455 is_mature=false
2023-08-15T09:19:29.340Z  INFO merge{merge_split_id=01H7W54FF69F8QM750A8EAD2QF split_ids=["01H7W48X2QS9P24GFQJJWGNRYG", "01H7W472CDW8CEXWSJZWS92DMH", "01H7W457QX8M45A5EDRSBNBDAZ", "01H7W43CP97BA5JKGFHRH8E198", "01H7W41HJ1EHKCZXW6G3KM9RJW", "01H7W3ZPJ9EV21M4FQ0MDDMHEJ"] typ=Merge}:uploader:upload{split=01H7W54FF69F8QM750A8EAD2QF}:store_split: quickwit_indexing::split_store::indexing_split_store: store-in-cache
thread 'tokio-runtime-worker' panicked at 'assertion failed: split_path.is_dir()', quickwit-indexing/src/split_store/local_split_store.rs:410:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2023-08-15T09:19:32.526Z  WARN merge{merge_split_id=01H7W4TP1QCMK6VRXR8GRCRYFF split_ids=["01H7W2935VHYVDR253V2YHX5HY", "01H7W1WJMGCTEEVSH21AJTYWZC", "01H7W0Y0EXTBHZZ721ABAQEE5R", "01H7W0CWMMB3ZSZNBGYH7N8GHS", "01H7VZYJ7X5XC8DAF1M53JYB44"] typ=Merge}:uploader: quickwit_indexing::actors::uploader: Failed to upload split. Killing! cause=Failed uploading key 01H7W4TP1QCMK6VRXR8GRCRYFF.split in bucket s3://gharchive-0.6-dev/indexes/gh-archive

Caused by:
    0: StorageError(kind=NotFound, source=No such file or directory (os error 2))
    1: No such file or directory (os error 2) split_id="01H7W4TP1QCMK6VRXR8GRCRYFF"
2023-08-15T09:19:32.527Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=MergePublisher-old-GPqO exit_status=Killed
2023-08-15T09:20:06.793Z  INFO quickwit_indexing::actors::indexing_service: Spawning indexing pipeline. pipeline_id=IndexingPipelineId { index_uid: IndexUid("gh-archive:01H7W7XJYVWVGM343K1M4G1W5E"), source_id: "_ingest-api-source", node_id: "quickwit-indexer-5", pipeline_ord: 0 }
2023-08-15T09:20:06.796Z  INFO spawn_merge_pipeline{index=gh-archive gen=0}: quickwit_indexing::actors::merge_pipeline: Spawning merge pipeline. index_id=gh-archive source_id=_ingest-api-source pipeline_ord=0 root_dir=/quickwit/qwdata/indexing/gh-archive%01H7W7XJYVWVGM343K1M4G1W5E%_ingest-api-source%0%T9T4TB merge_policy=StableLogMergePolicy { config: StableLogMergePolicyConfig { min_level_num_docs: 100000, merge_factor: 6, max_merge_factor: 12, maturation_period: 172800s }, split_num_docs_target: 10000000 }

Start at quickwit-indexing/src/split_store/local_split_store.rs:410 and trace how the REST index deletion propagates through the indexing service and its kill switches. Reproduce indexing concurrently with deletion, then verify the run avoids the panic and that a split uploaded to S3 after deletion is cleaned up.