[BUG] Hung peer recovery permanently blocks replica allocation.

Describe the bug

These were suggested by Codex after a multi hour analysis of pods

• Environment: OpenSearch 3.3.0, three-node cluster, default cluster.routing.allocation.node_concurrent_{incoming,outgoing} of 2.
• Steps: kill/restart a data node so its replicas need to recover; simulate a stuck recovery (e.g., block traffic between source and target). Observe that the shard
  remains INITIALIZING forever, consuming the only available recovery slot.
• Expected: OpenSearch eventually marks the recovery failed and retries / lets other replicas proceed.
• Actual: recovery stays stuck indefinitely, unassigned_shards never clear, operator must manually delete the shard or wipe the node.

Suggested fix:

1. Auto-fail hung recoveries: Add a timeout to peer recoveries so that if no bytes transfer within (say) X minutes, OpenSearch marks the shard copy as failed and retries
   on another node. This would free the node_concurrent_*_recoveries slots and let other replicas progress without manual intervention.
2. Heuristic for “no valid shard copy”: If a node rejoins with stale data and the cluster sees repeated NODE_LEFT → INITIALIZING loops, automatically wipe that shard
   copy (or at least flag it as unusable) so allocation can proceed. This mirrors what operators do manually by deleting PVCs.

Related component

No response

To Reproduce

There is no repro steps and it's hard to recover from such cases, would love to see a solution.

Expected behavior

The cluster recovers w/o manual intervention.

Additional Details

Additional details from another instance of the same on 2/10/2026.

Environment

• OpenSearch version: 3.3.0 (from GET /)
  Evidence file: cluster_info.json
• 3 data nodes

Symptoms

• Peer recoveries stuck in INIT for ~9.78–10.81 hours with 0 bytes/files recovered and translog.total = -1:
  • .plugins-ml-model-group – 10.81h in INIT
  • .plugins-ml-config – 10.81h in INIT
  • .opendistro_security – 10.81h in INIT
  • search-index.1770673512721 – 9.78h in INIT
    Evidence files: recovery_active_detailed.json, cat_recovery.txt

This is not just “slow”; it is effectively stalled.

Clear log messages
From delayed_allocation.log (extracted from opensearch-cluster-nodes-0.log):

[2026-02-10T10:47:33,510][INFO ][o.o.c.r.DelayedAllocationService] ... scheduling reroute for delayed shards in [59.7s] (8 delayed shards)
[2026-02-10T10:48:33,249][INFO ][o.o.c.r.DelayedAllocationService] ... scheduling reroute for delayed shards in [7.9m] (6 delayed shards)
[2026-02-10T11:50:54,571][INFO ][o.o.c.r.DelayedAllocationService] ... scheduling reroute for delayed shards in [9.9m] (2 delayed shards)
[2026-02-10T21:19:39,160][INFO ][o.o.c.r.DelayedAllocationService] ... scheduling reroute for delayed shards in [9.9m] (1 delayed shards)

These show the cluster repeatedly re‑routing delayed shards for 10+ hours without successful completion.

API evidence

• _recovery?detailed=true&active_only=true shows the four peer recoveries stuck in INIT for ~9–11 hours with 0 progress.
  Evidence file: recovery_active_detailed.json
• _cat/recovery shows the same in tabular form.
  Evidence file: cat_recovery.txt
• _cluster/health?level=shards shows ongoing unassigned/initializing replicas.
  Evidence file: cluster_health_shards.json
• cluster/allocation/explain shows allocation decisions throttled by recovery slot limits.
  Evidence files: allocation_explain*.json

Attachments

opensearch_recovery_evidence.zip

Start by reviewing the DelayedAllocationService log entries and the attached recovery_active_detailed.json, cat_recovery.txt, cluster_health_shards.json, and allocation_explain_*.json evidence. Reproduce or inspect the listed _recovery, _cat/recovery, _cluster/health, and _cluster/allocation/explain responses; done means stalled peer recoveries fail or retry, recovery slots are released, and replica allocation proceeds without manual shard or node cleanup.