StackExchange / StackExchange/StackExchange.Redis

HotKeysClusterTests.CanUseClusterFilter skips intermittently, hiding the assertions it exists for

Open Beginner friendly
#3,239 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
6.2k
Forks
1.6k
Avg merge
1d 15h
Merged PRs (30d)
43

Description

HotKeysClusterTests.CanUseClusterFilter skips intermittently, so it can stop testing anything without saying so.

Noticed while comparing suite skip counts across runs on an unrelated branch: the total moved between 149 and 151 for the same code. The difference is this test, and it is not environmental flakiness in the usual sense - it is a deliberate Skip on the result being empty.

Where

tests/StackExchange.Redis.Tests/HotKeysTests.cs:43

Assert.SkipWhen(result.CpuByKey.IsEmpty, "Expected at least one CPU result"); // can be weird in CI

Observed

Three consecutive local runs of just this test, against the docker cluster, no code changes between them:

Passed!  - Failed: 0, Passed: 4, Skipped: 0
Passed!  - Failed: 0, Passed: 4, Skipped: 0
  Skipped HotKeysClusterTests.CanUseClusterFilter(sample: True) (RESP3)
Passed!  - Failed: 0, Passed: 3, Skipped: 1

Only the sample: True case is affected, which fits: it runs with sampleRatio: 3 (line 25), so the server samples one request in three and can legitimately observe nothing in the window.

Why it matters

Everything the test exists to assert comes after that line - including Assert.Equal(sample, result.IsSampled) at line 62, which is the sampling behaviour the sample: true case is specifically about. So on the runs where sampling happens to catch nothing, the test reports success-by-skipping and checks none of it.

This is the same failure shape as the vacuous assertion fixed in #3226 and the gaps closed in #3235: a test that fails open is worse than no test, because the suite stays green and the count of what actually ran is not something anyone reads.

Suggested direction

Two parts, either or both:

  • Make the window deterministic enough that a result is guaranteed - e.g. drive enough traffic through the sampled slot that one-in-three cannot miss, rather than skipping when it does.
  • Failing that, split the assertions: the ones that do not depend on CpuByKey having entries (notably IsSampled) should run regardless, so a thin sampling window narrows what is checked rather than skipping the whole test.

If the skip really is unavoidable, it should at least be loud about which assertions were not reached.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tests/StackExchange.Redis.Tests/HotKeysTests.cs, especially the sampled setup near line 25, the skip at line 43, and the sampling assertion at line 62. Run HotKeysClusterTests.CanUseClusterFilter against the Docker cluster repeatedly. Done means the sample=true case no longer intermittently skips the assertions when CpuByKey is empty, while the sampling behavior remains checked.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, redis
Domain
testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.