[RFC] Enable test coverage infrastructure for end-2-end, functional and unittests
@sasdf is already working on this.
Since Jul 16, 2025.
- Dominant language
- SystemVerilog
- Stars
- 3.6k
- Forks
- 1.1k
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 141
Description
### Description
Test coverage metrics are an important software quality metric enabling the identification of test coverage gaps across, unit, functional and end-2-end test levels.
Coverage reports of released artifacts are also required for Common Criteria certification.
## Scope
This issue proposes the enablement of test coverage reports for the following components.
### Artifacts
* `ROM`
* `ROM_EXT`
* `cryptolib`
### Test Environments
* `unittest`
* `functest`
* `end-2-end`
* `cryptotest`
## Implementation Details
There two different approaches that can be followed to implement coverage reports:
1. Compile-time instrumentation
2. Trace generation from Simulation or Emulation environments.
Simulation traces are out of scope at the moment given performance limitations. `ROM` end-2-end test cases take hours to run in simulation environments.
Software based emulation (e.g. QEMU, Renode) are not an option at the moment due to lack of RTL/silicon feature parity.
The only reasonable option at the moment is to implement compile time instrumentation. This issue recommends using https://github.com/lowRISC/opentitan/pull/3080 as a reference for an implementation supported by Bazel.
## Tasks
- [ ] Implement test coverage instrumentation for `unittests`.
- [ ] Port https://github.com/lowRISC/opentitan/pull/3080 to Bazel to enable coverage for
`functest and `end-2-end`.
- [ ] Instrument the ROM to run from SRAM and with support for OTP emulation. This is to
enable `rom end-2-end` coverage testing.
- [ ] Enable OTP emulation for the `ROM_EXT`. Need to check if this is actually needed.
- [ ] Collect coverage metrics across test environments (i.e. `unittest`, `functest`, `end-2-end`.
- [ ] Generate aggregated coverage reports. The report should include git release information.
- [ ] Integrate coverage reports into nightly software regressions.
- [ ] Integrate coverage reports into release flows.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.