[coverage] Coverage Measurements

Creating this issue to capture the plan for coverage measurements. This will be a long journey, so we are taking a pragmatic approach by limiting the scope and intentionally including some temporary workarounds to unblock the functionality we need sooner. After this we can think about additional enhancements such as differential coverage for PRs and local changes, dashboard, and other quality of life enhancements.

- [ ] Coverage measurements
- [ ] On-target profile data
- [x] Initial experiments, PoC, and foundation (#3080)
- [ ] Enable coverage instrumentation using config settings and compiler flags as needed (#16947 , #16951)
- This is a temporary measure to unblock progress until we can integrate our changes with bazel. Some resources: [1](https://bazel.build/configure/coverage), [2](https://bazel.build/extending/rules#code_coverage), [3](https://bazel.build/rules/lib/InstrumentedFilesInfo).
- [x] Coverage runtime (#16762)
- [ ] Coverage instrumentation (#16944 , #16951)
- [x] Compiler flags for silicon creator targets (#16951)
- [x] Coverage measurements from functional tests (#16951)
- Measured coverage of (daily) commits between 2022-12-20 - 2022-12-31.
- [ ] Coverage measurements from ROM E2E tests
- This requires some investigation since instrumented binary does not fit in `32 KiB` when built with `-O0`. We **must** support coverage with optimization and be careful about what we are instrumenting.
- [ ] Reduce coverage instrumentation size overhead.
- [ ] Transfer profile buffer before ROM hands over execution (either to ROM_EXT or shutdown).
- [x] Coverage buffer transfer (#16762)
- Mostly there, we should be more clear about endianness, increase buffer size, and break long lines to prevent buffer issues.
- [x] Update vendored llvm_clang_rt_profile to llvmorg-13.0.1 (#16762)
- We collect raw profile data from our target and there are no cross-version compatibility guarantees.
- [x] Parse, index, and merge raw profile data in test logs (#16951)
- [x] Remove vendored repo and use bazel to pull llvm_clang_rt_profile (#16973)
- llvm-project releases a smaller archive that includes the part we need.
- [x] Off-target profile data (unit tests)
- ~Bazel should support this out of the box. We can use this to unblock CI integration and work on it in parallel.~
- ~Depending on the profile data format that bazel uses we may need to convert to/from llvm profile data.~
- Bazel has only primitive support for clang's source-based code coverage. We need to merge the profile data, process object files and generate reports.
- [x] bazel config changes for coverage measurements using clang (#16891)
- [x] build fixes for clang (#16890)
- [x] Add `clang-{lowRISC RISC-V toolchain version}` to `opentitan` container Dockerfile (#16888).
- lowRISC RISC-V toolchain version is 13.0.1.
- [x] Determine the libraries that we are interested in (mostly silicon_creator modules, drivers, and rom code). (#16891)
- [ ] Reports
- See [here](https://storage.googleapis.com/sw-coverage/index.html).
- [ ] On-target
- [x] Functests (silicon creator modules and drivers)
- [ ] ROM e2e tests
- Also consider gdb-based e2e tests. Find out if it's possible to measure coverage in asm files.
- [ ] Merged on-target coverage measurements
- [x] Off-target unit tests
- [x] Merged on-/off-target coverage measurements
- [ ] Additional columns in the merged report with data from unit/func/e2e.
- [ ] Additional reports such as mock and `*_unittest.cc` coverage.
- [x] ~Refactor current implementation for multiple coverage configs, e.g. queries in hjson logic in py.~
- I decided not to pursue this path in favor of better `bazel` integration because it turns into a "workflow orchestration" project which is essentially `bazel`.
- [x] Update landing page.
- [ ] Report generation and publication
- [x] Public GCS bucket
- `sw-coverage` under `opentitan.org/ot-fpga-runner` for now.
- [x] Bucket layout:
- `-//{merged.profiledata, merged.so, report.txt, html}`
- [x] Index page
- A [minimalist page](https://storage.googleapis.com/sw-coverage/index.html) ([gist](https://gist.github.com/4927af3452d10e04c66b16bf2ad50214)) for now.
- Should be updated as we update the layout of the GCS bucket to include additional report types.
- [x] Results table
- Dynamically generated.
- [x] Trend chart
- Added a line chart (chart.js). The [index page](https://storage.googleapis.com/sw-coverage/index.html) looks pretty nice now.
- [ ] Nightly runs
- We can start working on this sooner using off-target coverage measurements.
- [x] Local workstation
- Started running unit test coverage script and uploading results to GCS on 2022-12-21 (using `gsutil` for now).
- [ ] CI
- [ ] Build and Run
- [ ] Authenticate to GCP with OIDC/WIF and publish
- [ ] Improve coverage and coverage infrastructure
- [ ] Add dashboard pages to the repo
- [ ] Unit tests for rom.c (See #16022)
- [ ] False positives possible?
- What does `llvm-cov` do if we have the profile data but not the object file?
- [ ] `static inline` change required by optimizations eliminates hash-mismatch errors. Check if still need to create the merged libraries.
- [ ] Can we integrate with `bazel` and `bazel coverage` better?
- [ ] @dmcardle's [comment](https://github.com/lowRISC/opentitan/pull/16891#discussion_r1052555331)
- [ ] @dmcardle's [comment](https://github.com/lowRISC/opentitan/pull/16891#discussion_r1052531513)

cc @cfrantz @arunthomas @milesdai @drewmacrae @dmcardle

Start by reviewing the unchecked ROM end-to-end coverage and nightly CI items, along with the Bazel coverage resources linked in the issue. Inspect rom.c and the OpenTitan container Dockerfile where relevant. Done means the remaining selected coverage reports are generated, published, and available through the stated CI workflow.

Written by the indexing model from the issue text.