Coverage display on codecov.io does not match locally-generated HTML report
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 170
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 4
Description
I don't know whether this is a problem with grcov or Codecov, but you seem more accessible, so I'll start here.
I am attempting to set up coverage reporting of my Rust project using grcov and codecov.io. When I generate HTML coverage reports locally like so:
export RUSTFLAGS="${RUSTFLAGS:+$RUSTFLAGS }-Cinstrument-coverage"
cargo build
export LLVM_PROFILE_FILE="target/coverage/prof/%p-%m.profraw"
cargo test
grcov \
--source-dir . \
--binary-path target/debug \
--branch \
--excl-start 'mod tests \{' \
--ignore 'tests/*' \
-t html \
-o target/coverage/html \
target/coverage/prof
then the coverage is shown as 88.61%, and the report (mostly) makes sense. However, when I upload a report to codecov from GitHub Actions with the following workflow:
name: Test
on:
push:
pull_request:
schedule:
- cron: '0 0 1 * *'
jobs:
test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
toolchain:
- '1.60'
- stable
- beta
- nightly
steps:
- name: Check out repository
uses: actions/checkout@v3
- name: Configure Git identity
# Needed for a couple tests
run: |
git config --global user.name 'GitHub Actions'
git config --global user.email 'ghactions@nil.nil'
- name: Install Rust
run: |
rustup update ${{ matrix.toolchain }}
rustup default ${{ matrix.toolchain }}
rustup component add llvm-tools-preview
cargo install grcov
- name: Build crate
run: cargo build --verbose
env:
RUSTFLAGS: "-Cinstrument-coverage"
- name: Test crate
run: cargo test --verbose
env:
LLVM_PROFILE_FILE: "target/coverage/prof/%p-%m.profraw"
RUSTFLAGS: "-Cinstrument-coverage"
- name: Create coverage report
run: |
grcov \
--source-dir . \
--binary-path target/debug \
--branch \
--excl-start 'mod tests \{' \
--ignore 'tests/*' \
-t lcov \
-o lcov.info \
target/coverage/prof
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
files: lcov.info
fail_ci_if_error: false
then the coverage is shown as 48.21%, and much of the report does not match the HTML report, nor does it make much sense.
For example, compare this section of the HTML report:
with the corresponding section in codecov's report:
Another example: In the HTML report, all of the lines shown below are neither hit nor missed, yet codecov shows a mixture of hit & missed lines, including some of both in the comments!
In addition, I previously had my tests in my src/lib.rs file but later moved them to tests/. When they were in src/lib.rs, I was able to get the HTML report to show the tests as neither hit nor missed, yet Codecov showed the tests as largely missed, with some hits thrown in.
You can find the report uploaded to codecov for the "stable" run of this commit here: https://github.com/mozilla/grcov/files/9072729/build.txt.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the report using the local grcov commands and the GitHub Actions workflow, comparing the generated HTML report with lcov.info and the Codecov result. Start with grcov's Rust coverage collection and LCOV output, then determine whether the mismatch is produced by grcov or Codecov. Done means the discrepancy is reproduced and its source is identified with supporting report comparisons.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100