Boundary-value coverage: flag numeric comparisons whose boundary (left == right) is never tested
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 4.6k
- Forks
- 1.2k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 23
Description
Scenario
- JaCoCo version: current master (0.8.16-SNAPSHOT); the gap is independent of version
- Operating system: macOS (the proposal is OS-independent)
- Tool integration: API / CLI — the proof of concept is a standalone Java agent, validated against JaCoCo's own Maven tests and pgjdbc's Gradle tests
- Description of your use case: I want to know whether tests exercise the boundary value of a numeric comparison, not just both branches. For
if (i > 6), branch coverage reaches 100% as soon as somei <= 6and somei > 6run, even if no test ever usesi == 6— the off-by-one case. Today only mutation testing surfaces that, at a much higher cost. Reproducer (standalone agent, independent of core, changes no existing code): https://github.com/vlsi/jacoco/tree/boundary-value-coverage-poc/boundary-coverage-poc
Current Behaviour
Branch coverage reports each conditional as taken / not-taken, so it cannot distinguish "both sides ran" from "both sides ran and the boundary was tested". Even JaCoCo's own analysis code has lines that are fully branch-covered yet never exercise the boundary value:
MethodCoverageImpl#increment—branches.getTotalCount() > 1: tests exercise 0 and ≥ 2 branches, but never exactly 1, so the> 1vs>= 1boundary is untested.KotlinJvmOverloadsFilter#filter—methodNode.instructions.size() < 5: sizes below and above 5 are exercised, the threshold 5 is not.
Wanted Behaviour
An opt-in, experimental metric, kept separate from the existing counters, that for each ordered numeric comparison records which of three regions ran at run time: below the boundary (left < right), at it (left == right), and above it (left > right). The headline finding is "boundary missed": both ordered sides ran but == never did.
This stays within JaCoCo's bytecode-only model and needs no source parsing: it instruments IF_ICMP* ordered comparisons, ordered IF* against zero, and LCMP / FCMP* / DCMP* (whose result is already the three-way sign). I'm aware #1946, #1951 and #1504 were declined because source-level decision / condition coverage would require interpreting what the source treats as a decision; this proposal avoids that and reports only what the bytecode comparison itself reveals.
It is a heuristic with false positives, so the actionable signal is the "both sides run, boundary missed" subset; equality comparisons, operands inside && / ||, and structurally one-sided comparisons are skipped or demoted. Surfacing this as a real counter would touch the exec format, the probe model, and the report model, which is the part I'd want your read on. My question is whether a metric of this kind is something JaCoCo would consider in principle, or is firmly out of scope for core. If it's out of scope, I'll keep it as a standalone agent.
Frankly, I think the metric is cheap to compute, and it it provides high value input (low false positives), so having that information by default would be an overall improvement.
Possible Workarounds
Mutation testing already surfaces these gaps: PIT's CONDITIONALS_BOUNDARY (and the ROR operators) mutate </<=/>/>= and report a survivor when the boundary is untested. The cost is that the suite is re-run once per mutant.
The standalone agent above is a lighter workaround: it produces the same boundary signal from a single instrumented test pass. I cross-checked the two on pgjdbc over the same nine classes and the same tests:
- Every strong finding (both sides exercised, boundary never) sat on a surviving boundary mutant: 13 / 13 of the lines where PIT placed a mutant, with no contradictions.
- Across all flagged lines, 42 / 46 are PIT
SURVIVEDand 0 are PITKILLED, so no false positives against PIT's verdict. - The findings came from one instrumented test pass; PIT ran 1495 test executions (~9 per mutant) for nine classes and a single mutator.
Examples confirmed as surviving CONDITIONALS_BOUNDARY mutants:
ServerVersion#parseServerVersionStr—parts[1] > 99/parts[2] > 99: the limit value 99 is never fed in.LruCache.LimitedMap#removeEldestEntry—currentSize <= maxSizeBytes(anLCMP): the exact eviction threshold (currentSize == maxSizeBytes) is never tested.IntSet#add/#contains—value <= MAX_OID_TO_STORE_IN_BITSET: the bitset upper bound is never exercised, so the bitset / fallback-set switch at the boundary is untested.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin with the standalone boundary-coverage POC and the cited MethodCoverageImpl#increment and KotlinJvmOverloadsFilter#filter examples. Then trace how JaCoCo represents exec data, probes, and reports to assess the proposed impact. Done would require an agreed opt-in metric scope and acceptance decision for core, not merely a patch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- testing-qa, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100