Reconcile of large CHIs is much slower since 0.27: ActionPlan is re-rendered on every status update

What happened

After upgrading from 0.25.3 to 0.27.1, adding shards to a large (or even medium-sized) CHI became a lot slower (the larger the CHI, the more pronounced the difference).
Here are some comparisons from our test environment: one CHI with two clusters, scaling from 20 to 39 shards, 2 replicas each (80 -> 156 hosts total).

• 0.25.3: ~27 min
• 0.27.1: ~1h41m, operator CPU pinned at its limit for the whole run, and ~3000 "got error, will retry" status update conflict logs per run (vs ~250 on 0.25.3)

Cause

After looking into, we identified the ActionPlan stringification to be the culprit.
Since the action plan was added to the status storage ConfigMap (0.27.x), buildStatusResourceData calls Status.ActionPlan.String() on every status update attempt. Each call re-renders the whole diff: it iterates the diff maps (random order, so the same plan produces different bytes every time) and Dump()s every diffed object — the full value is rendered and then truncated to 256 chars. For a scale-up the added items are entire shards, so every render dumps every added shard.

Status updates run several times per host, plus once per conflict retry, so that's thousands of renders per reconcile. In 0.25.x the plan was rendered exactly once per reconcile (LogActionPlan) and wasn't part of status writes at all.

It also feeds back on itself: the render makes each status update slower, which widens the window between reading the CR and writing the status, which causes more resourceVersion conflicts, which cause 1s-sleep retries, and every retry renders the plan again.

Fix

Rendering the plan once at construction (see PR) restores 0.25.x behavior while keeping the storage feature. With that one change on top of 0.28.0, the same test scale-up takes 18–21 min, conflict retries are back to ~350, and operator CPU is back to 0.25.3 levels.

Start at buildStatusResourceData and inspect how Status.ActionPlan.String() is used during status writes, alongside LogActionPlan and the ActionPlan construction path. Done means the plan is rendered once at construction while remaining in status storage; compare reconcile duration, conflict retries, and operator CPU with the reported results.