Cross-subnet control plane migration causes full unrecoverable cluster outage under default load balancer

Summary

Migrating control plane nodes to a destination network that is not Layer 2 adjacent to the source causes a complete, unrecoverable cluster outage under the default (OpenShiftManagedDefault) load balancer, not just loss of external API access. This was reproduced live end to end on a real cluster (OCP 5.0.0-rc.1, vSphere IPI) under OCPSTRAT-2677 hackathon testing.

What was tested

A full migration from a source vCenter to a destination vCenter on a different subnet (confirmed non-overlapping CIDRs, no shared L2), using the default load balancer configuration. The migration was allowed to run to the point where all three control plane nodes had moved to the destination subnet.

What happened

At the moment the third and final control plane node completed its move to the destination subnet:

• oc against the external API endpoint failed outright: dial tcp <api-vip>:6443: connect: no route to host, not a timeout, the address was completely unclaimed on the network.
• Confirmed directly on a live node that api-int.<cluster-domain> and api.<cluster-domain> resolve to the identical single virtual IP address in this load balancer topology (keepalived.conf on the node shows exactly two VRRP instances, one for API, one for Ingress, no separate internal-only VIP).
• Every node in the cluster, including the original workers that never left the source subnet, stopped renewing their Lease objects in kube-node-lease at the same moment, and began reporting Ready=Unknown.
• Core platform pods across openshift-apiserver, openshift-authentication, openshift-console, the default Ingress router, and openshift-monitoring all went Pending with nowhere schedulable.
• This is not a transient blip. Once every control plane node is on the destination subnet, no node can ever reclaim the original VIP via VRRP again, since VRRP requires Layer 2 adjacency (gratuitous ARP), which does not cross a subnet boundary.

Why this happens

The operator's control plane migration logic (internal/controller/, docs/plans/control-plane-migration-coverage-plan.md) creates new worker MachineSets on the target failure domain, then updates the ControlPlaneMachineSet in place and delegates the actual rolling replacement entirely to the upstream CPMS operator. There is no reference anywhere in the operator to VIPs, keepalived, load balancer type, or machineNetworks, and no check for Layer 2 adjacency between source and destination failure domains.

The default load balancer implementation on vSphere/baremetal IPI floats the API and Ingress virtual IP addresses among control plane/router-hosting nodes via VRRP, which fundamentally cannot work once every node providing it has moved off the original network segment. OpenShift's own baremetal IPI documentation already establishes this exact constraint for install-time multi-subnet control plane topologies (ipi-install-establishing-communication-between-subnets.adoc): a UserManaged load balancer is required for control plane nodes spanning multiple subnets. This migration operator does not carry that same requirement forward as a Day 2 precondition.

Severity and customer impact

This ticket already has multiple linked customer accounts and at least one linked priority escalation describing a customer needing to migrate dozens of production clusters off VMware infrastructure. Any customer using the default load balancer (the out-of-the-box choice for vSphere/baremetal IPI) whose destination network is not the same Layer 2 segment as the source will walk directly into this outage with no warning, and (see the related issue on load balancer type immutability) currently has no way to recover the cluster afterward short of a full reinstall.

Recommendation

The operator should add a hard, blocking preflight check that refuses to begin a control plane rolling replacement unless the source and destination failure domains are confirmed to be on the same Layer 2 segment, or Infrastructure.status.platformStatus.vsphere.loadBalancer.type is already UserManaged. Given the outage is total and currently has no path to recovery, this cannot be a "proceed with a warning" preflight item, it needs to hard block.

Environment

• OCP 5.0.0-rc.1, vSphere IPI, loadBalancer.type: OpenShiftManagedDefault
• vcf-migration-operator dev-preview build

Filed from hackathon testing under OCPSTRAT-2677.

Start in internal/controller/ and read docs/plans/control-plane-migration-coverage-plan.md to trace how failure domains and ControlPlaneMachineSet updates are handled. Identify the preflight point for checking load balancer type and Layer 2 adjacency, then verify that cross-subnet migration is blocked unless the load balancer is UserManaged or the networks are compatible.