openshift / openshift/vcf-migration-operator
No validation of the customer UserManaged load balancer; misconfiguration hangs migration silently and indefinitely
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1
- Forks
- 6
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 12
Description
Summary
The operator does not validate the customer's own UserManaged load balancer configuration before or during a migration. When a customer's load balancer has a routing gap on one of its three independent paths (API, machine-config-server, Ingress), the operator gives no diagnostic signal at all, the migration CR's status just repeats a generic, unchanging progress message while a node hangs indefinitely.
Why this matters
Per the discussion on #109 and the related load-balancer-type-immutability issue, UserManaged is effectively the only viable load balancer choice for any customer whose destination network is not Layer 2 adjacent to the source, which is exactly the scenario this migration feature exists to serve. In practice this means real customers will be hand-configuring or hand-maintaining an external load balancer themselves, often for the first time, under the time pressure of an active migration.
What was observed
During hackathon testing, two independent, unrelated load balancer misconfigurations were introduced by hand while operating a real HAProxy instance as the migration's UserManaged load balancer, and both produced the identical failure signature: a silent, indefinite hang with nothing surfaced anywhere in oc get vmwarecloudfoundationmigration or oc describe.
- A missing backend route for the machine-config-server port (22623) on the load balancer meant a new control plane replacement node never got an IP, never started guest tools, and
kubelet/crionever attempted to start, confirmed directly via hypervisor console access. The migration CR's status the entire time read onlyWorkloadMigrated=False: Control plane rolling out (2/4 updated, 4/4 ready), an unchanging, generic message. - The Ingress backend pool on the load balancer was never updated to include newly created target-subnet workers. Once the original workers were drained and deleted at the end of the migration, the console and all application routes became unreachable, while
WorkloadMigratedandSourceCleanedboth reportedTruethe entire time.
In both cases, the underlying vSphere control plane and machine-api layers were completely healthy, the entire failure was invisible to the operator because it never independently checks whether its own three required load balancer paths are actually working.
Recommendation
- Provide an explicit, customer-runnable pre-migration validation step (or a preflight check inside the operator itself) that independently tests the API (6443), machine-config-server (22623), and Ingress (80/443) paths through the customer's configured load balancer, rather than assuming all three are correctly and independently configured.
- Detect a node that has not registered after an extended period and surface it as an explicit condition, for example
WorkloadMigrated=False: node <name> has not registered after N minutes, check load balancer connectivity to port 22623, instead of a generic unchanging "rolling out" message. - Document explicitly that a customer using
UserManagedmust keep their load balancer's backend pools (API, machine-config-server, and Ingress independently) in sync with each control plane and worker replacement throughout the migration, and specifically re-verify Ingress backend membership once old workers are removed at the end.
Environment
- OCP 5.0.0-rc.1, vSphere,
loadBalancer.type: UserManaged vcf-migration-operatordev-preview build
Filed from hackathon testing under OCPSTRAT-2677.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files or tests. Start by locating UserManaged load-balancer handling and migration status or condition updates; inspect how the API (6443), machine-config-server (22623), and Ingress (80/443) paths are represented. Done means misconfiguration is detected and surfaced before or during migration, with the required customer guidance documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100