Flaky: mcp-oidc-config deletion tests race the controller's 30s deletion requeue
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 300
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 184
Description
Summary
The deletion specs in the mcp-oidc-config integration suite fail intermittently on CI. The failure looks structural rather than random: the controller rechecks a blocked deletion on a 30s requeue, and the suite's Eventually timeout is also 30s, so a spec passes only if reconcile lands inside whatever margin is left. Loaded runners lose that race.
Evidence
Run 30574241215, three attempts against the same commit:
| Attempt | Result | Spec that failed |
|---|---|---|
| 1 | action_required | - |
| 2 | failure | should be deleted after MCPRemoteProxy reference is removed |
| 3 | failure | should be deleted after VirtualMCPServer reference is removed |
A different sibling failed on each attempt, both timing out at exactly 30.000s. The same suite passes locally.
Mechanism
MCPOIDCConfig deletion is blocked while workloads still reference it, and rechecked on a fixed requeue:
// cmd/thv-operator/controllers/mcpoidcconfig_controller.go:241
// Requeue to check again later
return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
The controller watches only its own type, so deleting a referencing workload wakes nothing:
// cmd/thv-operator/controllers/mcpoidcconfig_controller.go:360
return ctrl.NewControllerManagedBy(mgr).
For(&mcpv1beta1.MCPOIDCConfig{}).
Complete(r)
The tests use a matching budget:
// cmd/thv-operator/test-integration/mcp-oidc-config/mcpoidcconfig_controller_integration_test.go:18
timeout = time.Second * 30
interval = time.Millisecond * 250
Each spec deletes the referencing workload and then waits timeout for the config to disappear. Because nothing wakes the config, it is only rechecked at the 30s tick, which starts slightly before the Eventually clock. The remaining margin is whatever reconcile latency fits in that gap.
Three specs share this shape. Two have been observed failing:
mcpoidcconfig_mcpremoteproxy_integration_test.go:454(failed, attempt 2)mcpoidcconfig_virtualmcpserver_integration_test.go:325(failed, attempt 3)mcpoidcconfig_mcpserver_integration_test.go:269("should be deleted after references are removed", same mechanism, not yet observed failing)
Suggested fix
Raising the suite timeout above the requeue interval (60s) resolves it with a one-line change.
A structural fix would be to watch the three referencing types and map back to the config, so removing a reference wakes the deletion check immediately instead of waiting for the poll. That overlaps with #5607, and might be better handled there.
Related
#5607 (Simplify config-CRD reference tracking). Distinct issue, but adjacent. The field indexers proposed there are already implemented in SetupWithManager.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the timeout constants in cmd/thv-operator/test-integration/mcp-oidc-config/mcpoidcconfig_controller_integration_test.go and the deletion specs in the MCPRemoteProxy, VirtualMCPServer, and MCPServer integration test files. Run the affected integration suite and confirm all three reference-removal deletion specs pass reliably despite the controller's 30-second requeue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, testing-qa
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 86/100