bytecodealliance / bytecodealliance/wasmtime

cranelift: line coverage gaps in ISLE-generated lowering

Open
#7,549 3 comments 3 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
18.6k
Forks
1.8k
Avg merge
1d 18h
Merged PRs (30d)
126

Description

I was musing about whether it would be possible for us to generate inputs to ISLE lowering that would exercise all code paths. So it occurred to me to check what coverage the cranelift unit/integration tests already have.

The line coverage reports suggest high coverage, but there are some gaps that might be worth discussion.

## Coverage Generation

I found the [`taiki-e/cargo-llvm-cov`](https://github.com/taiki-e/cargo-llvm-cov) tool to be very easy to use for coverage report generation.

In the `cranelift` directory:

```
cargo llvm-cov --disable-default-ignore-filename-regex --ignore-filename-regex='cargo/' --html
```

By default the `llvm-cov` tool won't report coverage for the ISLE generated files. The `--disable-default-ignore-filename-regex` was necessary to disable this default behavior, and then `--ignore-filename-regex='cargo/'` ignores coverage from external dependencies.

## Results

High-level coverage metrics for the ISLE generated files (columns are function/line/region):

![Screen Shot 2023-11-15 at 8 00 46 PM](https://github.com/bytecodealliance/wasmtime/assets/7133685/35a186da-045a-4847-9d16-cc8c2d04dbe3)

There's a lot of uncovered code to sift through, a lot of which is clearly not a concern (including uncovered `unreachable!` statements). But there seem to be some cases in here that might be worth looking into. For example, in `constructor_to_amode_add`:

![Screen Shot 2023-11-15 at 8 02 50 PM](https://github.com/bytecodealliance/wasmtime/assets/7133685/3111469c-fa32-4678-9b71-931ccd722bfd)

## Questions

* Are any of these coverage gaps interesting, and worth adding tests for?
* Are these gaps covered by fuzzing? (related #6293)
* Is 100% line coverage actually a reasonable goal here? What is the right metric? (related #1151)

cc @jameysharp @fitzgen

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.