stacklok / stacklok/toolhive

Flaky: mcp-oidc-config deletion tests race the controller's 30s deletion requeue

Open Beginner friendly
#6,155 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

good-first-issue kubernetes
Dominant language
Go
Stars
2.2k
Forks
300
Avg merge
1d 15h
Merged PRs (30d)
184

Description

Summary

The deletion specs in the mcp-oidc-config integration suite fail intermittently on CI. The failure looks structural rather than random: the controller rechecks a blocked deletion on a 30s requeue, and the suite's Eventually timeout is also 30s, so a spec passes only if reconcile lands inside whatever margin is left. Loaded runners lose that race.

Evidence

Run 30574241215, three attempts against the same commit:

Attempt Result Spec that failed
1 action_required -
2 failure should be deleted after MCPRemoteProxy reference is removed
3 failure should be deleted after VirtualMCPServer reference is removed

A different sibling failed on each attempt, both timing out at exactly 30.000s. The same suite passes locally.

Mechanism

MCPOIDCConfig deletion is blocked while workloads still reference it, and rechecked on a fixed requeue:

// cmd/thv-operator/controllers/mcpoidcconfig_controller.go:241
// Requeue to check again later
return ctrl.Result{RequeueAfter: 30 * time.Second}, nil

The controller watches only its own type, so deleting a referencing workload wakes nothing:

// cmd/thv-operator/controllers/mcpoidcconfig_controller.go:360
return ctrl.NewControllerManagedBy(mgr).
    For(&mcpv1beta1.MCPOIDCConfig{}).
    Complete(r)

The tests use a matching budget:

// cmd/thv-operator/test-integration/mcp-oidc-config/mcpoidcconfig_controller_integration_test.go:18
timeout  = time.Second * 30
interval = time.Millisecond * 250

Each spec deletes the referencing workload and then waits timeout for the config to disappear. Because nothing wakes the config, it is only rechecked at the 30s tick, which starts slightly before the Eventually clock. The remaining margin is whatever reconcile latency fits in that gap.

Three specs share this shape. Two have been observed failing:

  • mcpoidcconfig_mcpremoteproxy_integration_test.go:454 (failed, attempt 2)
  • mcpoidcconfig_virtualmcpserver_integration_test.go:325 (failed, attempt 3)
  • mcpoidcconfig_mcpserver_integration_test.go:269 ("should be deleted after references are removed", same mechanism, not yet observed failing)

Suggested fix

Raising the suite timeout above the requeue interval (60s) resolves it with a one-line change.

A structural fix would be to watch the three referencing types and map back to the config, so removing a reference wakes the deletion check immediately instead of waiting for the poll. That overlaps with #5607, and might be better handled there.

Related

#5607 (Simplify config-CRD reference tracking). Distinct issue, but adjacent. The field indexers proposed there are already implemented in SetupWithManager.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the timeout constants in cmd/thv-operator/test-integration/mcp-oidc-config/mcpoidcconfig_controller_integration_test.go and the deletion specs in the MCPRemoteProxy, VirtualMCPServer, and MCPServer integration test files. Run the affected integration suite and confirm all three reference-removal deletion specs pass reliably despite the controller's 30-second requeue.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, testing-qa
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
86/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.