lowRISC / lowRISC/opentitan

[RFC] Enable test coverage infrastructure for end-2-end, functional and unittests

Open
#26,510 4 comments 0 reactions 1 assignee View on GitHub

@sasdf is already working on this.

Since Jul 16, 2025.

RFC:Proposal
Dominant language
SystemVerilog
Stars
3.6k
Forks
1.1k
Avg merge
2d 22h
Merged PRs (30d)
141

Description

### Description

Test coverage metrics are an important software quality metric enabling the identification of test coverage gaps across, unit, functional and end-2-end test levels.

Coverage reports of released artifacts are also required for Common Criteria certification.

## Scope

This issue proposes the enablement of test coverage reports for the following components.

### Artifacts

* `ROM`
* `ROM_EXT`
* `cryptolib`

### Test Environments

* `unittest`
* `functest`
* `end-2-end`
* `cryptotest`

## Implementation Details

There two different approaches that can be followed to implement coverage reports:

1. Compile-time instrumentation
2. Trace generation from Simulation or Emulation environments.

Simulation traces are out of scope at the moment given performance limitations. `ROM` end-2-end test cases take hours to run in simulation environments.

Software based emulation (e.g. QEMU, Renode) are not an option at the moment due to lack of RTL/silicon feature parity.

The only reasonable option at the moment is to implement compile time instrumentation. This issue recommends using https://github.com/lowRISC/opentitan/pull/3080 as a reference for an implementation supported by Bazel.

## Tasks

- [ ] Implement test coverage instrumentation for `unittests`.
- [ ] Port https://github.com/lowRISC/opentitan/pull/3080 to Bazel to enable coverage for
`functest and `end-2-end`.
- [ ] Instrument the ROM to run from SRAM and with support for OTP emulation. This is to
enable `rom end-2-end` coverage testing.
- [ ] Enable OTP emulation for the `ROM_EXT`. Need to check if this is actually needed.
- [ ] Collect coverage metrics across test environments (i.e. `unittest`, `functest`, `end-2-end`.
- [ ] Generate aggregated coverage reports. The report should include git release information.
- [ ] Integrate coverage reports into nightly software regressions.
- [ ] Integrate coverage reports into release flows.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.