bytecodealliance / bytecodealliance/wasmtime
Move towards "test the exact release artifacts" in CI
- Dominant language
- Rust
- Stars
- 18.6k
- Forks
- 1.8k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 126
Description
In yesterday's Wasmtime meeting we had a discussion about testing, CI, and the artifacts that we produce. Currently the actual released artifacts are not actually tested in CI at all, they're simply produced. This isn't a great situation and ideally we would test the exact artifacts that CI produces rather than a different build configuration.
There's a few challenges to tackle when handling this though:
* Most of the test suite is not architected to run the `wasmtime` CLI or the `libwasmtime.so` C API. Some of `tests/all/cli_tests.rs` runs the `wasmtime` executable, but the executable is produced as part of `cargo test` and isn't a released artifact. The C API is only tested through the C examples (so very lightly).
* Testing artifacts might require a different command to execute or new infrastructure other than `cargo test`. It's unclear what this infrastructure would look like.
* Testing "all possible configurations" isn't feasible, even with the generous concurrency that we already have. For example we can't test the matrix of {platform, opt/debug, debug-asserts/no, crate} in CI as that would be too much.
We didn't reach many firm conclusions in the meeting but I wanted to open a follow-up issue to track further progress. A few points we did identify were:
* We should probably have one CI job test with `--release` to test with optimizations (just one platform though, probably not all of them)
* We should ideally run at least a "smoke test" for released artifacts.
cc @tschneidereit
Contributor guide
Assessment
This issue has not been assessed yet.