jacoco / jacoco/jacoco

Boundary-value coverage: flag numeric comparisons whose boundary (left == right) is never tested

Open
#2,190 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type: enhancement
Dominant language
Java
Stars
4.6k
Forks
1.2k
Avg merge
1d 9h
Merged PRs (30d)
23

Description

Scenario
  • JaCoCo version: current master (0.8.16-SNAPSHOT); the gap is independent of version
  • Operating system: macOS (the proposal is OS-independent)
  • Tool integration: API / CLI — the proof of concept is a standalone Java agent, validated against JaCoCo's own Maven tests and pgjdbc's Gradle tests
  • Description of your use case: I want to know whether tests exercise the boundary value of a numeric comparison, not just both branches. For if (i > 6), branch coverage reaches 100% as soon as some i <= 6 and some i > 6 run, even if no test ever uses i == 6 — the off-by-one case. Today only mutation testing surfaces that, at a much higher cost. Reproducer (standalone agent, independent of core, changes no existing code): https://github.com/vlsi/jacoco/tree/boundary-value-coverage-poc/boundary-coverage-poc
Current Behaviour

Branch coverage reports each conditional as taken / not-taken, so it cannot distinguish "both sides ran" from "both sides ran and the boundary was tested". Even JaCoCo's own analysis code has lines that are fully branch-covered yet never exercise the boundary value:

  • MethodCoverageImpl#incrementbranches.getTotalCount() > 1: tests exercise 0 and ≥ 2 branches, but never exactly 1, so the > 1 vs >= 1 boundary is untested.
  • KotlinJvmOverloadsFilter#filtermethodNode.instructions.size() < 5: sizes below and above 5 are exercised, the threshold 5 is not.
Wanted Behaviour

An opt-in, experimental metric, kept separate from the existing counters, that for each ordered numeric comparison records which of three regions ran at run time: below the boundary (left < right), at it (left == right), and above it (left > right). The headline finding is "boundary missed": both ordered sides ran but == never did.

This stays within JaCoCo's bytecode-only model and needs no source parsing: it instruments IF_ICMP* ordered comparisons, ordered IF* against zero, and LCMP / FCMP* / DCMP* (whose result is already the three-way sign). I'm aware #1946, #1951 and #1504 were declined because source-level decision / condition coverage would require interpreting what the source treats as a decision; this proposal avoids that and reports only what the bytecode comparison itself reveals.

It is a heuristic with false positives, so the actionable signal is the "both sides run, boundary missed" subset; equality comparisons, operands inside && / ||, and structurally one-sided comparisons are skipped or demoted. Surfacing this as a real counter would touch the exec format, the probe model, and the report model, which is the part I'd want your read on. My question is whether a metric of this kind is something JaCoCo would consider in principle, or is firmly out of scope for core. If it's out of scope, I'll keep it as a standalone agent.

Frankly, I think the metric is cheap to compute, and it it provides high value input (low false positives), so having that information by default would be an overall improvement.

Possible Workarounds

Mutation testing already surfaces these gaps: PIT's CONDITIONALS_BOUNDARY (and the ROR operators) mutate </<=/>/>= and report a survivor when the boundary is untested. The cost is that the suite is re-run once per mutant.

The standalone agent above is a lighter workaround: it produces the same boundary signal from a single instrumented test pass. I cross-checked the two on pgjdbc over the same nine classes and the same tests:

  • Every strong finding (both sides exercised, boundary never) sat on a surviving boundary mutant: 13 / 13 of the lines where PIT placed a mutant, with no contradictions.
  • Across all flagged lines, 42 / 46 are PIT SURVIVED and 0 are PIT KILLED, so no false positives against PIT's verdict.
  • The findings came from one instrumented test pass; PIT ran 1495 test executions (~9 per mutant) for nine classes and a single mutator.

Examples confirmed as surviving CONDITIONALS_BOUNDARY mutants:

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with the standalone boundary-coverage POC and the cited MethodCoverageImpl#increment and KotlinJvmOverloadsFilter#filter examples. Then trace how JaCoCo represents exec data, probes, and reports to assess the proposed impact. Done would require an agreed opt-in metric scope and acceptance decision for core, not merely a patch.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing-qa, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.