downloaded_splits directory is not being cleaned up in the indexer

In the indexer, the downloaded-splits directory is not being cleaned up, causing disk usage to repeatedly spike to 100% and then drop back down.

Below are the disk usage metrics and the actual disk usage details.

• Disk Usage

• Directory Info

I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata$ du -sh *
158G    indexer-split-cache
108G    indexing
16K     lost+found
12K     queues
74M     wal
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata$ cd index
indexer-split-cache/ indexing/
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata$ cd indexing/
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata/indexing$ du -sh *
4.0K    cloudfront-analytics%01K46QJFB8TFWFXQG2BHWTX4ZT%_ingest-api-source%01K5QXPJVJJYJ24KB4NJT316J7%yWOHyx
5.0M    cloudfront-analytics%01K46QJFB8TFWFXQG2BHWTX4ZT%cloudfront-analytics%01K5QXTDX291C6K4XH4X0H85XB%Z8nMgZ
4.0K    log.common.access_log_v2_quickwit%01JWA8T4D80XNZ0JXVFJBBZGDP%_ingest-api-source%01K5QXPJVJ8PAPG6QKXDG0FC4Z%3yotTw
4.0K    log.common.accesslog_v1_1%01JNN5K4NT0MX73J1W62HNPBF3%_ingest-api-source%01K5QXPJVJ7ARDY3T6WZSJHXZN%UMjIUL
4.0K    log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%_ingest-api-source%01K5QXPJVJMA4DBW1792YPS8TE%DnBA3h
108G    log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX22SWVX87D3TFZ21CF%E9Gchn
27M     log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX2562N8QFK96Q0YNFJ%H8bor1
33M     log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX2MQ502KPRGRVVERH1%xSZOTa
26M     log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX2P2VPGW0B82CDHDH8%fFyBAF
4.0K    log.common.karrot_audit_authorization_log_v1%01K1ARAG165KH1BK7118FCSSHR%_ingest-api-source%01K5QXPJVJ0FYDGGBHN5FF4Y2F%DvB40E
4.0K    log.common.karrot_audit_authorization_log_v1%01K1ARAG165KH1BK7118FCSSHR%kafka_to_karrot_audit_authorization_log_v1_ca%01K5QXTDX2PRK0JXBBCHSEEJB6%1JFoxJ
4.0K    log.common.karrot_audit_log_v1%01K1ARAC9ZVJ5AA28D9K6575TY%_ingest-api-source%01K5QXPJVJ88QP0712KMTHXRWY%YP8bBZ
4.0K    log.karrot-gateway.access_log_v2_quickwit%01K4Y6016Q0QWFT41GC5ZVEM4X%_ingest-api-source%01K5QXPJVJFYRW8AV3SKFD142Y%7ho0S8
4.0K    log.karrotgateway.accesslog_v1_1%01JPAG00RG5TRD0F3X0Z1ZGPTC%_ingest-api-source%01K5QXPJVJS8MCS7VBGWHB404X%prNb4Y
4.0K    otel-logs-v0_7%01JE5QBDXD8GPEAQR2GZKB5RRR%_ingest-api-source%01K5QXPJVJGJ4QS9ZHVGNY8465%UmuipQ
4.0K    otel-logs-v0_9%01JY22ACBKBTMT2SFSDS23ZT6Y%_ingest-api-source%01K5QXPJVJFJE1H87WK18HR248%zyxdJz
4.0K    otel-traces-v0_7%01JNG4MET50159AFDBMJ9A4EJ3%_ingest-api-source%01K5QXPJVJF2H9QZFWZ93DBXVB%CwwVk3
4.0K    otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-api-source%01K5QXPJVJFMGGDXGZFAFG4GDB%gAUQYX
4.0K    otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-source%01K5RJZSJ542K53K84DVM58B61%hfrZpG
4.0K    otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-source%01K5RK6FJ0X9AEFRAT2J1YC83V%jwYe7Q
4.0K    otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-source%01K5RMNTX1VB9SXD1942DDC4Q7%MOCBOH
4.0K    otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-source%01K5RN3BS6NCWX78MP7A2SFZSZ%OQOQM3
4.0K    otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-source%01K5T1AWJZH974XYB52MH2YDCQ%5hmdMy
4.0K    otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-source%01K5T2SWW5CH4457XAY40ENF5G%2ptknm
20K     otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-source%01K5T3C4WY0W8J9BZ4CFPJ8Y6Q%zgliAv
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata/indexing$ cd log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX22SWVX87D3TFZ21CF%E9Gchn
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata/indexing/log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX22SWVX87D3TFZ21CF%E9Gchn$ du -sh *
108G    merge%MKKSPc
504K    split-01K5T5FM1G8YN36DVMCQBBFGMH-UHdGlE
34M     split-01K5T5FM2WTS7NY81CGTK7801K-jyDT2u
12K     split-01K5T5FM53GEZ9JYENXSSJEW1Z-zO2jBt
12K     split-01K5T5FNZT8HSD4NA9JW4YJZWZ-xmx8YX
12K     split-01K5T5FSSYVYWHNV3SFY6SREVS-rBhVvP
12K     split-01K5T5FVERCJH32XYEMTF5V1AQ-BV1cL9
12K     split-01K5T5GQBKASBZB518JXEJFSKC-TPApuk
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata/indexing/log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX22SWVX87D3TFZ21CF%E9Gchn$ cd merge%MKKSPc/
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata/indexing/log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX22SWVX87D3TFZ21CF%E9Gchn/merge%MKKSPc$ du -sh *
108G    downloaded-splits%u5FqAB
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata/indexing/log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX22SWVX87D3TFZ21CF%E9Gchn/merge%MKKSPc$ cd downloaded-splits%u5FqAB/
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata/indexing/log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX22SWVX87D3TFZ21CF%E9Gchn/merge%MKKSPc/downloaded-splits%u5FqAB$ du -sh *
15G     01K5R3ZGEAYSG1APWW1TFPJNXV.split
27G     01K5R6WJS6BCZ83MTT0QQVD215.split
23G     01K5R9NEZE01B3RQM2Z3CVP05R.split
22G     01K5RCHDHXHXHKMYQXRG9GRKZV.split
24G     01K5RFH9T91R08DAE8B5RXEB1G.split

The issue names no source files or tests. Start by tracing the indexer's merge workflow and the lifecycle of the downloaded-splits directory, using the reported directory layout to reproduce the disk growth. Done means completed or abandoned downloads are cleaned up and the indexer no longer leaves large split files behind.