control-plane: recover from a non-converging indexing scheduler (stuck "reapply last plan" loop)

Summary

The control plane can get stuck indefinitely in a non-converging indexing-scheduler loop, stalling ingestion for the affected shards until manual restart of indexers.

This issue tracks the staged work to make the loop observable and then self-recovering.

Background

control_running_plan compares what indexers report running with the last applied plan:

• Different nodes -> recompute the plan from the model (rebuild_plan).
• Same nodes, different tasks -> re-send the same plan, assuming an indexer just has not applied it yet.

That assumption breaks when the plan cannot be applied. It assigns a shard that no longer exists (our case) in the indexer's WAL (a phantom Open shard after a crash/restart onto a fresh volume). The indexer can never run that task, so has_same_tasks() stays false forever and the loop re-sends the same plan every ~30s. Reapplying cannot fix a plan that is wrong, it only re-sends it over and over again.

Today this is also invisible: the loop logs the same info line each cycle, so a one-cycle propagation lag is indistinguishable from a cluster stuck for hours, and there is nothing to alert on.

Fix Plan

Stage 1. Observability - #6585

Stage 2. Recovery - #6602 (includes #6585 but can be merged separately, on top)

Start by reading the control_running_plan path, including rebuild_plan and has_same_tasks(), then review the staged work in #6585 and #6602. Done means the non-converging loop is observable and the planned recovery prevents repeated reapplication from stalling ingestion.