openshift / openshift/oadp-operator
e2e: route-reachability check has no retry, flakes when router hasn't propagated the restored route yet
@Joeavaikath is already working on this.
Since Aug 13, 2026.
- Dominant language
- Go
- Stars
- 92
- Forks
- 93
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 43
Description
Found while triaging a CI failure on #2373 (unrelated to that PR's actual change — a NodeAgent DaemonSet reconcile fix).
What happened
ci/prow/4.23-e2e-test-aws failed on the "MySQL application two Vol CSI" case:
Attempt 1 actual error at backup_restore_suite_test.go:288: "Error getting pod for the proxy command: no Pod found"
Backup and restore both completed successfully (36 items restored, both EBS CSI snapshots restored fine). The app pod was confirmed running (MariaDB ready, API server up) seconds before the check ran. The failure was purely in post-restore route-reachability verification.
Root cause
tests/e2e/backup_restore_suite_test.go calls lib.VerifyBackupRestoreData(...) (tests/e2e/lib/apps.go:446), which tries GetRouteEndpointURL first, and — if that fails — falls back to finding a pod via GetFirstPodByLabel(kubeClient, namespace, "curl-tool=true") (tests/e2e/lib/apps.go:536) to proxy the request instead. Neither path retries:
GetRouteEndpointURLfailing (route not yet propagated by the OpenShift router) is treated as immediately fatal for that path rather than tried again after a short wait.- The curl-tool proxy-pod fallback returns
"Error getting pod for the proxy command: no Pod found"(tests/e2e/lib/apps.go:538) as soon asGetFirstPodByLabeldoesn't find it once — no wait for that pod to become schedulable/ready either.
Since neither the primary route check nor the fallback proxy-pod lookup has any retry/backoff, a transient window where the route hasn't propagated yet (a normal, expected delay after a restore) fails the whole verification outright.
Suggested fix
Wrap the route-reachability check (and/or the proxy-pod lookup fallback) in a gomega.Eventually(...) with a reasonable timeout (a few minutes) and poll interval, matching the pattern already used elsewhere in this test suite (e.g. gomega.Eventually(lib.IsNamespaceDeleted(...), time.Minute*5, time.Second*5) in the same file).
[!Note]
Responses generated with Claude
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.