Old VitessShards remain idle=Unknown after Reshard complete removes source topology, blocking cleanup

Summary

After completing a reshard from 32 to 64 shards and removing the old 32-shard partitioning from the keyspace configuration, obsolete Kubernetes VitessShard resources remain with status.idle: "Unknown". The operator refuses to delete them even though an old source shard's topology record is already missing and the supplied serving topology references only the new layout.

Please address this cleanup failure without requiring users to patch controller-owned status or manually delete shard resources, while preserving serving-topology safety checks.

Steps to reproduce

Reported sequence:

1. Start with a keyspace using a 32-shard partitioning.
2. Reshard to 64 shards and run Reshard complete.
3. Remove the old 32-shard partitioning from the applied keyspace configuration, expecting the operator to clean up its source shards.
4. Inspect remaining VitessShard resources and their status.idle.
5. Query an old source shard with vtctldclient GetShard <keyspace>/<old-shard-range> and inspect serving topology using GetSrvKeyspaces.

The exact workflow command flags and timing were not captured in this report; the sequence has not been independently reproduced in a test environment.

Actual behavior and evidence

The operator reports:

refusing to delete unwanted VitessShard ...: The shard can't be turned down because it's potentially in the serving set. You must migrate all served types in all cells to another shard before removing this shard.

• Remaining old shard resources show status.idle: "Unknown".
• GetShard for an old source shard returns node doesn't exist.
• vtctldclient GetSrvKeyspaces <keyspace> returned a single cell with all three serving partitions using the new 64-shard layout:

Cell	Served type	References
cell_name	1 — PRIMARY	-04, 04-08, 08-0c, …, fc-
cell_name	2 — REPLICA	Same 64-shard layout
cell_name	3 — RDONLY	Same 64-shard layout

The old equal-32 ranges (-08, 08-10, …, f8-) are absent from the supplied partitions. This evidence covers the returned a single name cell; it is not an independent verification of every possible cell.

Expected behavior

Once the old partitioning is no longer desired and the operator can safely establish that no cell/tablet type serves the old shards, cleanup should converge automatically—even if reshard completion has already removed the global shard topology records.

A missing topology record should not leave obsolete resources permanently blocked solely because the existing idleness check requires that record. Conversely, missing records or topology read failures must not be treated as unconditional permission to delete potentially serving shards.

Suspected cause / relevant code

Source inspection at commit 769469d985b125a27efdba484e6b5c2729ac95d7 suggests the following path. This commit is the inspected repository revision, not a confirmed deployed operator version.

1. ReconcileVitessShard.Reconcile resets status before recomputing it, so an earlier idle observation is not retained.
2. reconcileTopology only assigns Status.Idle after both GetShard and GetShardServingCells succeed. A missing shard record takes the TopoGetFailed / failed to get shard info: ... branch and requeues instead. That event text is identified from code, not supplied runtime logs.
3. The keyspace controller's PrepareForTurndown requires Status.Idle == ConditionTrue. An unknown value blocks deletion with the serving-set warning.

This is consistent with source topology records disappearing before Kubernetes cleanup completes. The precise deletion timing and actor have not been independently confirmed from logs.

Requested fix and regression coverage

• Handle confirmed missing shard records during obsolete-shard cleanup using a safe way to validate serving references across all relevant cells and tablet types.
• Distinguish confirmed absence from connection errors, timeouts, permission failures, and incomplete topology observations; continue blocking unsafe or unverifiable cleanup.
• Improve diagnostics to distinguish a missing topology record from an actual remaining serving reference.
• Add regression coverage for reshard completion removing source topology before the old partitioning is removed from the configuration.
• Cover repeated reconciliations/status resets so cleanup does not depend on catching a transient idle: "True" value.
• Verify that a remaining serving reference in any cell or tablet type still prevents deletion, including when the global shard record is missing.

Workaround considered

Manually patching the status subresource to the string "True" was discussed as a way to pass the deletion guard. It is race-dependent because the shard controller recomputes status, can authorize destructive cleanup, and is not an appropriate normal resharding procedure. Successful cleanup using this workaround has not been confirmed in this report.

Environment

• Resource API: planetscale.com/v2, VitessShard.
• Resharding: 32 source shards to 64 target shards.
• Serving topology supplied for the single cell
• Operator image/version, Vitess version, and Kubernetes version: not provided.

Related report

https://github.com/planetscale/vitess-operator/issues/169 describes old shards remaining after DropSources and removal of the old partitioning. It is closed; this report does not establish whether the underlying cause is identical.
https://github.com/planetscale/vitess-operator/issues/741

Start with ReconcileVitessShard.Reconcile in pkg/controller/vitessshard/vitessshard_controller.go, then trace reconcileTopology in pkg/controller/vitessshard/reconcile_topo.go and PrepareForTurndown in pkg/controller/vitesskeyspace/reconcile_shards.go. Add regression coverage for missing shard records after reshard completion and repeated status resets. Done means cleanup proceeds only when all relevant cells and tablet types are verified not to serve the old shards, while failures and incomplete observations still block deletion.