StackExchange / StackExchange/StackExchange.Redis
HotKeysClusterTests.CanUseClusterFilter skips intermittently, hiding the assertions it exists for
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 6.2k
- Forks
- 1.6k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 43
Description
HotKeysClusterTests.CanUseClusterFilter skips intermittently, so it can stop testing anything without saying so.
Noticed while comparing suite skip counts across runs on an unrelated branch: the total moved between 149 and 151 for the same code. The difference is this test, and it is not environmental flakiness in the usual sense - it is a deliberate Skip on the result being empty.
Where
tests/StackExchange.Redis.Tests/HotKeysTests.cs:43
Assert.SkipWhen(result.CpuByKey.IsEmpty, "Expected at least one CPU result"); // can be weird in CI
Observed
Three consecutive local runs of just this test, against the docker cluster, no code changes between them:
Passed! - Failed: 0, Passed: 4, Skipped: 0
Passed! - Failed: 0, Passed: 4, Skipped: 0
Skipped HotKeysClusterTests.CanUseClusterFilter(sample: True) (RESP3)
Passed! - Failed: 0, Passed: 3, Skipped: 1
Only the sample: True case is affected, which fits: it runs with sampleRatio: 3 (line 25), so the server samples one request in three and can legitimately observe nothing in the window.
Why it matters
Everything the test exists to assert comes after that line - including Assert.Equal(sample, result.IsSampled) at line 62, which is the sampling behaviour the sample: true case is specifically about. So on the runs where sampling happens to catch nothing, the test reports success-by-skipping and checks none of it.
This is the same failure shape as the vacuous assertion fixed in #3226 and the gaps closed in #3235: a test that fails open is worse than no test, because the suite stays green and the count of what actually ran is not something anyone reads.
Suggested direction
Two parts, either or both:
- Make the window deterministic enough that a result is guaranteed - e.g. drive enough traffic through the sampled slot that one-in-three cannot miss, rather than skipping when it does.
- Failing that, split the assertions: the ones that do not depend on
CpuByKeyhaving entries (notablyIsSampled) should run regardless, so a thin sampling window narrows what is checked rather than skipping the whole test.
If the skip really is unavoidable, it should at least be loud about which assertions were not reached.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in tests/StackExchange.Redis.Tests/HotKeysTests.cs, especially the sampled setup near line 25, the skip at line 43, and the sampling assertion at line 62. Run HotKeysClusterTests.CanUseClusterFilter against the Docker cluster repeatedly. Done means the sample=true case no longer intermittently skips the assertions when CpuByKey is empty, while the sampling behavior remains checked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, redis
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 76/100