CAPZ deletion deadlock: availability-prober blocks capi-provider when the guest API is permanently unavailable

Summary

HostedCluster deletion on Azure can deadlock permanently when the guest API server
is unreachable for the lifetime of the cluster (e.g. the customer resource group or the
etcd KMS Key Vault has been deleted). This was confirmed in an incident (internal tracker
AROSLSRE-1167, Prow run 2057959692289708032, 2026-05-22) that required manual finalizer
removal to recover, and was manually remediated on 2026-06-11 20:49 UTC.

Root cause chain

1. The capi-provider (CAPZ) Deployment has an availability-prober init container
   injected as InitContainers[0], gating pod startup on the guest KAS /readyz
   endpoint. The prober loop has no overall timeout or retry limit — it blocks forever
   until it gets a 2XX response.
   • control-plane-operator/controllers/hostedcontrolplane/v2/capi_provider/component.go
   • support/controlplane-component/defaults.go
   • support/podspec/containers.go (AvailabilityProber)
2. If the guest KAS never comes back (customer resource group deleted, or the etcd KMS Key
   Vault deleted so etcd can't decrypt and KAS can't serve), the init container never
   completes, so the CAPZ manager container never starts.
3. Because CAPZ never runs, it never processes the AzureMachine's finalizer
   (cluster-api-provider-azure's MachineFinalizer) and never sets a Ready=False / Reason=DeletionFailed condition on it.
4. HostedClusterReconciler.delete in the hypershift-operator waits for the CAPI Cluster
   to be deleted, which in turn waits on its Machine/AzureMachine children
   (hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go, delete()).
5. An existing escape hatch, Azure.DeleteOrphanedMachines
   (hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure.go), force
   -removes the AzureMachine finalizer for machines stuck in deletion — but only when the
   DeletionFailed condition is present. Since CAPZ never ran, that condition is never set,
   so the hatch is silently skipped, and HostedCluster deletion never finishes without
   manual finalizer removal.

Immediate mitigation (this repo)

A follow-up change broadens Azure.DeleteOrphanedMachines to also orphan a stuck
AzureMachine when the capi-provider Deployment itself is unavailable (no available
replicas, or missing) for longer than the existing 10-minute deletionFailedThreshold,
independent of the DeletionFailed condition. This unblocks deletion for the case that
triggered the incident (customer resource group already deleted, so the underlying Azure
VMs are already gone) without requiring CAPZ to ever start.

Remaining/root-cause work (this issue)

The mitigation above is intentionally narrow: it does not let CAPZ attempt real cloud
cleanup when the underlying infrastructure (e.g. just the KMS Key Vault, not the whole
resource group) is still intact — in that scenario, orphaning the AzureMachine leaves the
real Azure VM running (a resource leak), because nothing ever asks Azure to delete it.

A more complete fix should address the underlying deadlock design:

• Consider excluding/bypassing the availability-prober gate on the capi-provider (and
  other CAPI provider) deployments while the owning HostedCluster/HostedControlPlane is
  being deleted, so CAPZ can start and attempt real deletion via the Azure API even when the
  guest KAS is down.
• Alternatively, add a bounded timeout/deadline to the availability-prober itself so a
  permanently-unreachable target does not block a container indefinitely, with a way for
  deletion-critical components to opt out of the current infinite-retry semantics.
• Ensure whichever approach is chosen degrades safely: it must not let deletion proceed by
  orphaning machines whose backing infrastructure could still be reachable and cleanly
  deleted.

Related

• hypershift-operator/controllers/hostedcluster/internal/platform/aws/aws.go
  (DeleteOrphanedMachines) has a similar but more aggressive analogue for AWS (clears
  finalizers unconditionally on invalid credentials, no DeletionFailed/threshold
  dependency) — worth comparing when designing the broader fix.

Trace deletion through hostedcluster_controller.go, internal/platform/azure/azure.go, and the CAPZ deployment and availability-prober entry points named in the issue. Compare the AWS DeleteOrphanedMachines behavior and determine how deletion-critical components should behave when the guest API is unavailable. Done means HostedCluster deletion can recover from a permanently unavailable guest API without orphaning machines whose Azure infrastructure remains reachable.