openshift / openshift/vcf-migration-operator

No validation of the customer UserManaged load balancer; misconfiguration hangs migration silently and indefinitely

Open
#113 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1
Forks
6
Avg merge
2d 20h
Merged PRs (30d)
12

Description

Summary

The operator does not validate the customer's own UserManaged load balancer configuration before or during a migration. When a customer's load balancer has a routing gap on one of its three independent paths (API, machine-config-server, Ingress), the operator gives no diagnostic signal at all, the migration CR's status just repeats a generic, unchanging progress message while a node hangs indefinitely.

Why this matters

Per the discussion on #109 and the related load-balancer-type-immutability issue, UserManaged is effectively the only viable load balancer choice for any customer whose destination network is not Layer 2 adjacent to the source, which is exactly the scenario this migration feature exists to serve. In practice this means real customers will be hand-configuring or hand-maintaining an external load balancer themselves, often for the first time, under the time pressure of an active migration.

What was observed

During hackathon testing, two independent, unrelated load balancer misconfigurations were introduced by hand while operating a real HAProxy instance as the migration's UserManaged load balancer, and both produced the identical failure signature: a silent, indefinite hang with nothing surfaced anywhere in oc get vmwarecloudfoundationmigration or oc describe.

  1. A missing backend route for the machine-config-server port (22623) on the load balancer meant a new control plane replacement node never got an IP, never started guest tools, and kubelet/crio never attempted to start, confirmed directly via hypervisor console access. The migration CR's status the entire time read only WorkloadMigrated=False: Control plane rolling out (2/4 updated, 4/4 ready), an unchanging, generic message.
  2. The Ingress backend pool on the load balancer was never updated to include newly created target-subnet workers. Once the original workers were drained and deleted at the end of the migration, the console and all application routes became unreachable, while WorkloadMigrated and SourceCleaned both reported True the entire time.

In both cases, the underlying vSphere control plane and machine-api layers were completely healthy, the entire failure was invisible to the operator because it never independently checks whether its own three required load balancer paths are actually working.

Recommendation

  1. Provide an explicit, customer-runnable pre-migration validation step (or a preflight check inside the operator itself) that independently tests the API (6443), machine-config-server (22623), and Ingress (80/443) paths through the customer's configured load balancer, rather than assuming all three are correctly and independently configured.
  2. Detect a node that has not registered after an extended period and surface it as an explicit condition, for example WorkloadMigrated=False: node <name> has not registered after N minutes, check load balancer connectivity to port 22623, instead of a generic unchanging "rolling out" message.
  3. Document explicitly that a customer using UserManaged must keep their load balancer's backend pools (API, machine-config-server, and Ingress independently) in sync with each control plane and worker replacement throughout the migration, and specifically re-verify Ingress backend membership once old workers are removed at the end.

Environment

  • OCP 5.0.0-rc.1, vSphere, loadBalancer.type: UserManaged
  • vcf-migration-operator dev-preview build

Filed from hackathon testing under OCPSTRAT-2677.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files or tests. Start by locating UserManaged load-balancer handling and migration status or condition updates; inspect how the API (6443), machine-config-server (22623), and Ingress (80/443) paths are represented. Done means misconfiguration is detected and surfaced before or during migration, with the required customer guidance documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.