rossoctl / rossoctl/context-guru

flaky: two load-related test flakes in the suite that gates every deploy

Open
#91 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
54
Forks
22
Avg merge
20h 42m
Merged PRs (30d)
89

Description

Two intermittent failures, both surfacing only under concurrent load. One investigation, not two mysteries found separately six weeks apart.

Neither is caused by the code under test. Both surfaced while a dozen agents ran proxies on one box at load average 10–14 — a condition normal CI does not reproduce, which is why they have gone unnoticed.

1. dash/TestEventsDerivesItsScopeFromTheResolver

Fails on a full go test ./... competing with another concurrent run.

  • 5/5 clean isolated, 30/30 clean under -count=30 -cpu=1, green on a solo package run.
  • Reproduced on a commit whose only dash change was a comment, so it cannot reach /api/events.
  • Cause: timing-sensitive SSE test — it asserts on whatever the recorder has flushed at the moment it looks. The dash package takes ~356s, so under contention the flush it needs has not landed.

2. TestDashboardAddsNoRequestLatencyWithContentCapture

Established by interleaved runs against origin/main, so it predates and is independent of any current branch.

  • Cause: a latency assertion measured on a machine already saturated. It compares request latency with content capture on against a budget that assumes the box is not competing with a dozen proxies.

Suggested fix

Both want an explicit synchronisation point rather than an implicit one:

  • For (1): poll for the expected event with a deadline, the way wait_for_rows already does for the request table, instead of asserting on the first read.
  • For (2): the budget needs to be either load-relative or the measurement taken under a quiescence check. Raising the constant just moves the flake.

Sleeping longer fixes neither.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Find the tests named dash/TestEventsDerivesItsScopeFromTheResolver and TestDashboardAddsNoRequestLatencyWithContentCapture, then inspect the existing wait_for_rows synchronization used by the request table. Reproduce with concurrent or repeated runs, and verify both tests remain deterministic under contention without relying on longer sleeps or simply raising the latency constant.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
performance, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
54/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.