BrighterCommand / BrighterCommand/Fences
Coverage collection is unreliable, and blocks the coverlet 10 upgrade
- Dominant language
- C#
- Stars
- 60
- Forks
- 1
- Avg merge
- 22h 50m
- Merged PRs (30d)
- 5
Description
## Summary
How we collect coverage is unreliable, and it is now blocking a dependency update. There are two symptoms with one underlying subject — **how branch and line coverage are gathered and counted** — so they are worth solving together rather than separately.
Neither symptom is a real loss of test coverage. In every case below the tests themselves passed.
## Symptom 1 — coverlet 6.0.4 drops coverage data non-deterministically
`coverlet.msbuild` 6.0.4 intermittently reports coverage below threshold, or fails outright, on **one target framework while the other two report the correct figure for the same tests on the same commit**. It has surfaced four times in the last week, across three different test projects and both Linux and Windows:
| Run | OS | Project | Symptom |
| --- | --- | --- | --- |
| [32841065953](https://github.com/BrighterCommand/Fences/actions/runs/32841065953) | ubuntu | `Specs` | net9.0 reported **0% / 0% / 0%**; net10.0 and net8.0 both reported 94.74 / 95.06 / 91.98 |
| [32841065953](https://github.com/BrighterCommand/Fences/actions/runs/32841065953) | windows | `Core.Tests` | `Unable to read beyond the end of the stream` on net9.0 and net8.0 |
| [33154114062](https://github.com/BrighterCommand/Fences/actions/runs/33154114062) | windows | `Extensions.Tests` | `Unable to read beyond the end of the stream` on net9.0; threshold failure on net8.0 |
| [33262613610](https://github.com/BrighterCommand/Fences/actions/runs/33262613610) | ubuntu | `Core.Tests` | net9.0 reported **99.95 / 99.81**; net10.0 and net8.0 both reported 100 / 100 |
Every one cleared on `gh run rerun --failed` with no code change.
The last row is the concerning one. The earlier failures announced themselves as obviously bogus — 0%, or a stream exception. **99.95% does not.** It reads exactly like somebody genuinely missed a line, and the only thing distinguishing it from a real regression is that two other frameworks reported 100% for identical tests. That is a subtle tell to rely on, and it is the sort of thing that eventually gets "fixed" by quietly lowering a threshold.
It is also not free: it failed the `9.0.0` release build and needed a manual re-run before `publish-nuget` would go.
## Symptom 2 — coverlet 10.0.1 counts branches differently, so we cannot take the update
#6 bumps `coverlet.msbuild` from 6.0.4 to 10.0.1 and fails:
```
error : The minimum branch coverage is below the specified 100
[test/Paramore.Fences.Core.Tests::TargetFramework=net10.0]
```
This one is **not** a flake — it reproduces, and it is a real behavioural change: coverlet 10 identifies branches the 6.x line did not. So we are pinned to a version that flakes, and the upgrade that might fix the flakiness is gated behind deciding what our branch-coverage numbers should actually be.
## Why this is one piece of work
Taking #6 means answering "is 100% branch coverage on `Paramore.Fences.Core` still the right gate, under a tool that counts branches differently?" That is a coverage-policy question, not a dependency bump, and it should not be settled as a side effect of merging a Dependabot PR.
## Where the configuration lives
- `eng/Test.targets` — `CollectCoverage`, `CoverletOutputFormat`, `ExcludeByAttribute`, ReportGenerator wiring
- `test/Paramore.Fences.Core.Tests/*.csproj`, `Extensions.Tests`, `RateLimiting.Tests`, `Testing.Tests` — `100`
- `test/Paramore.Fences.Specs/*.csproj` — `94,94,91`
## Worth investigating
Not proposals, just the threads that look most promising:
- **`CollectCoverage=true` uses coverlet's MSBuild in-process collector.** The `coverlet.collector` datacollector (`--collect:"XPlat Code Coverage"`) is generally the more robust path, and the in-process collector writing per-framework output during a multi-targeted build is a plausible source of the truncated/partial results above. Switching how we *pick up* coverage may make symptom 1 disappear regardless of version.
- Whether the three target frameworks are racing over shared coverage output paths.
- What coverlet 10 actually counts that 6.x did not on `Core.Tests`, and whether those branches deserve tests or an exclusion.
- Whether a hard `100` threshold is the right instrument, given it makes any collection wobble a build failure.
## Acceptance
- Coverage figures are reproducible across all three target frameworks for the same commit.
- #6 can be merged, or closed with a recorded reason.
- No threshold is lowered merely to accommodate a collection defect.
Contributor guide
Research direction
Read eng/Test.targets and the listed test project files to trace coverage collection, output paths, and thresholds; compare the failing workflow runs across all three target frameworks. Investigate the collector and coverlet 10 branch-counting behavior, then verify that coverage is reproducible, the #6 update can be merged or has a recorded reason, and no threshold was lowered for a collection defect.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, github-actions
- Domain
- build-system, ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100