Flaky: optimizer search-quality E2E spec is blocking unrelated PRs

Summary

VirtualMCPServer Optimizer Multi-Backend [It] should return semantically relevant results (search quality) (test/e2e/thv-operator/virtualmcp/virtualmcp_optimizer_multibackend_test.go:304) fails nondeterministically and is currently failing E2E Test Lifecycle on PRs whose diffs cannot possibly affect it.

Observed today: 4 failures in 5 runs, across three unrelated branches — two comment-only changes (#6017, #6025) and one test-only change (#6021). On one commit (17598b9b) it failed, was re-run, and failed again, while the other two kind versions passed the identical commit.

Why it is hard to recognise

Three things conspire to make this look like infrastructure rather than a test:

1. The job exits with a bare exit code 201 and no test-level annotation, so the GitHub check surface shows nothing actionable. 201 is Ginkgo's interrupted exit code, not a test-failure code.
2. Three failures are reported, but only one is real. The other two are [INTERRUPTED] collateral from Ginkgo aborting sibling parallel processes:

   [FAIL]        Optimizer Multi-Backend [It] should return semantically relevant results (search quality)
   [INTERRUPTED] Yardstick Base ... should discover a new backend when added to the group
   [INTERRUPTED] Redis-Backed Session Sharing ... [BeforeAll] cross-pod restore
   Ran 142 of 142 Specs — 139 Passed | 3 Failed
   FAIL! - Interrupted by Other Ginkgo Process
   

   Chasing either interrupted spec is a dead end.
3. Because the two interrupted specs differ run to run, the failure looks like it moves around.

Why the spec is fragile

It asserts semantic ranking: for each of four natural-language queries, find_tool must return a tool whose name contains an expected substring.

{query: "repeat or echo back a message",                     expectMatch: "echo",         backend: "yardstick"},
{query: "retrieve content from a web page or URL",           expectMatch: "fetch",        backend: "gofetch"},
{query: "check security vulnerabilities in open source packages", expectMatch: "vulnerability", backend: "osv"},
{query: "create a pull request on a code repository",        expectMatch: "pull_request", backend: "github"},

That runs against a corpus aggregated from 11 backends (backend1Name…backend11Name: yardstick, fetch, osv, github, terraform, playwright, puppeteer, memory, everything, ida-pro-mcp, pagerduty). The assertion therefore depends on embedding similarity and top-N truncation across hundreds of competing tools — whether the expected tool lands inside the returned window is not a deterministic property. It is additionally timing-sensitive: which backends have completed discovery when the query runs changes the corpus size and therefore the ranking.

So this is asserting a quality property with a correctness assertion.

Unexplained: all four failures were on kindest/node:v1.34.3

Every failure landed on v1.34.3; v1.33.7 and v1.35.1 have not failed once today, including on the same commits. I have no mechanism to offer for that — each version runs in its own job on its own runner, and the suite is identical. With n=4 it may be coincidence, or it may point at resource contention affecting the embedding container on that job. Flagging it as an observation rather than a diagnosis; worth checking whether it holds over a larger sample.

Suggested directions

Not prescribing a fix, since the right call depends on what the spec is meant to guarantee:

• Assert a weaker, deterministic property — e.g. that results are non-empty and drawn from the expected backend, rather than that a specific tool name ranks in the window.
• Shrink the corpus for this spec — run search-quality against a small, fixed backend set where ranking is stable, and leave the 11-backend configuration to the specs that are actually about multi-backend aggregation.
• Mark it as a quality signal, not a gate — if the assertion is inherently probabilistic, it may belong outside the merge-blocking suite.
• Independently: the exit code 201 opacity is worth addressing so a Ginkgo interruption surfaces the originating failure in the check annotation. That would have saved most of the investigation here.

Evidence

Generated with Claude Code

Start with test/e2e/thv-operator/virtualmcp/virtualmcp_optimizer_multibackend_test.go:304 and the E2E Test Lifecycle configuration that runs it. Reproduce the flaky search-quality spec across the listed kind node versions, then determine which proposed direction fits the intended guarantee. Done means the semantic-ranking check no longer nondeterministically blocks unrelated PRs and its failures expose the originating cause.