mozilla / mozilla/grcov

More detail/documentation for "functions" coverage

Open
#971 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
1.4k
Forks
170
Avg merge
3d 10h
Merged PRs (30d)
4

Description

This is a bit of a wishy washy issue, but it comes down to: I saw a big red number on my otherwise-green coverage report, but I still don't really understand what it means or how to get more detail on it (or how to fix it). As I go into more detail I've flagged the more specific questions and issues I have.

There's an example project here, and here's the coverage report. It's from a real Rust project I'm working on, it's deliberately not minimal (I'll get to that), and the command to generate the coverage report is:

env \
LLVM_PROFILE_FILE="test-binary-%p-%m.profraw" \
RUSTFLAGS="-C instrument-coverage" \
RUST_BACKTRACE=full \
cargo test

grcov . \
-t html \
--binary-path ./target/debug -s . \
--branch --llvm \
--ignore-not-existing --keep-only 'src/**' --ignore 'src/bin/test_*' \
--excl-br-start "mod tests \{" --excl-start "mod tests \{" \
--excl-br-line "#\[derive\(" --excl-line "#\[derive\(" \
-o coverage

When I generate the report, it shows the line and branch coverage as 100%. But between those two is something labelled "functions" which sits at 52.27% (in RED). Now, with no other information, I'd guess that that means only about 50% of the functions are being tested. But several things make me think this guess is wrong:

  • How could line coverage be 100% if function coverage is only 50%?
  • Drilling down into the code view shows every line to be green. (Question: I assume this is because that view only shows line coverage? Or not? Is there a way to switch between the three metrics?)
  • I did search for questions/issues around this, and found #476 and $454. But both those issues say to try the "new" (in 2020) source-based coverage. I'm pretty sure I'm using that now. (Right?)
  • Those issues say that this might be related to macros. But (issue:) the report does not show me what macros, either by highlighting them in the place they're used or by listing them separately.
  • This is why I haven't made a minimal example: if I reduce this project down to a single macro, it's easy to see the problem with grcov's analysis but it's also hard to see the problem with its reporting. There it is! I might say. That's the macro causing all the trouble! But try doing that with this project and you'll quickly run out of patience.

Issue: Documentation would remove the guesswork, but I cannot find details on this in Grcov's docs.

My overall issue is simply this: if low function coverage warrants a big, red number on the top of the report, it should also provide a way to account for that number, near to where it's shown. A link to documentation would be better than nothing, but ideally a way to visually display why the number is what it is.

Environment details

Gitlab CI: rust:1-slim docker image

$ cargo --version
cargo 1.67.1 (8ecd4f20a 2023-01-10)
$ rustc --version
rustc 1.67.1 (d5a82bbd2 2023-02-07)
$ grcov --version
grcov 0.8.13

Dev machine: Ubuntu 22.10

~ ⚬ cargo --version
cargo 1.67.1 (8ecd4f20a 2023-01-10)
~ ⚬ rustc --version
rustc 1.67.1 (d5a82bbd2 2023-02-07)
~ ⚬ grcov --version
grcov 0.8.13

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked grcov-example project, its generated coverage report, and the documented command using --branch and --llvm. Clarify what the functions, lines, and branches metrics mean, how macros affect the report, and what documentation or report links would let users account for the displayed function coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
documentation, testing-qa
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.