[BUG] Snapshot - deletion fails due to "Duplicate key ... (attempted merging values)..."

Describe the bug

We use the snapshot feature to backup system and kibana indices once a day and store them on an on-prem S3 using the repository-s3 plugin.
We're now in the situation that the s3 storage ran out of space and the snapshot repository is in an inconsistent state. The deletion of the kibana indices fails with the exception shown below. Creating new snapshots works.

opensearch-prod-data-2 opensearch {"type": "console", "timestamp": "2025-03-04T11:51:26,514Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "opensearch-prod", "node.name": "opensearch-prod-data-2", "message": "path: /_snapshot/kibana-indices/test-after-storage-increase, params: {pretty=true, repository=kibana-indices, snapshot=test-after-storage-increase}", "cluster.uuid": "Aln5wNjuRKKJvV8aNkW7Tg", "node.id": "kJP7Rx2USDGvXuB9sf2jIg" , 
opensearch-prod-data-2 opensearch "stacktrace": ["org.opensearch.transport.RemoteTransportException: [opensearch-prod-master-2][192.168.37.69:9300][cluster:admin/snapshot/delete]",
opensearch-prod-data-2 opensearch "Caused by: org.opensearch.repositories.RepositoryException: [kibana-indices] failed to delete snapshots [test-after-storage-increase/IYR0KL41QrWMugFr7bmzEw]",
opensearch-prod-data-2 opensearch "at org.opensearch.repositories.blobstore.BlobStoreRepository$2.onFailure(BlobStoreRepository.java:1075) ~[opensearch-2.18.0.jar:2.18.0]",
opensearch-prod-data-2 opensearch "at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:993) ~[opensearch-2.18.0.jar:2.18.0]",
opensearch-prod-data-2 opensearch "at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:54) ~[opensearch-2.18.0.jar:2.18.0]",
opensearch-prod-data-2 opensearch "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.lang.Thread.run(Thread.java:1583) [?:?]",
opensearch-prod-data-2 opensearch "Caused by: java.lang.IllegalStateException: Duplicate key EppnePWWS52J9uyjDbK6Fg (attempted merging values org.opensearch.repositories.s3.S3BlobContainer@1adc053c and org.opensearch.repositories.s3.S3BlobContainer@733b6f83)",
opensearch-prod-data-2 opensearch "at java.util.stream.Collectors.duplicateKeyException(Collectors.java:135) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:182) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]",
opensearch-prod-data-2 opensearch "at org.opensearch.repositories.s3.S3BlobContainer.children(S3BlobContainer.java:566) ~[?:?]",
opensearch-prod-data-2 opensearch "at org.opensearch.repositories.blobstore.BlobStoreRepository$2.doRun(BlobStoreRepository.java:1056) ~[opensearch-2.18.0.jar:2.18.0]",
opensearch-prod-data-2 opensearch "at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1005) ~[opensearch-2.18.0.jar:2.18.0]",
opensearch-prod-data-2 opensearch "at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.18.0.jar:2.18.0]",
opensearch-prod-data-2 opensearch "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]",
opensearch-prod-data-2 opensearch "at java.lang.Thread.run(Thread.java:1583) ~[?:?]"] }

Deleting the objects with duplicated key that is stated in the exception did not change the situation. It will just show the next duplicated key and so on and so on.

The issue occured with the snapshot of the 25.02. A lot of those duplicated keys got that modification date. Unfortunately exactly this snapshot is NOT listed, even not as failed.
During the time when s3 was out of space I tried to create several snapshots manually, they popped up for a second but then were simply vanished from the list of snapshots.

Related component

Storage:Snapshots

To Reproduce

1. Create snapshot repository from type S3 with a limited in space
2. Setup snapshot policy to backup '.kibana_*' indices daily
3. Get the s3 in state with no more space left
4. Let the snapshot "fail", maybe for a few days
5. Add storage to s3
6. Snapshots should be created again
7. Try to delete snapshots

Expected behavior

Failed snapshots are shown and can be deleted although they failed.
Snapshot repository fixes the inconsistency on its own.

Additional Details

Plugins

• Default plugins shipped with docker image
• repository-s3

Additional context
OpenSearch version: 2.18.0

Start with org.opensearch.repositories.s3.S3BlobContainer.children at S3BlobContainer.java:566 and the deletion path at BlobStoreRepository.java:1056, using the duplicate-key stack trace as the entry point. Reproduce with an S3 repository that runs out of space during snapshot creation, then verify that failed snapshots remain listed and can be deleted after storage is restored.