bytecodealliance / bytecodealliance/wasmtime

Move towards "test the exact release artifacts" in CI

Open
#9,881 0 comments 0 reactions 0 assignees View on GitHub
ci
Dominant language
Rust
Stars
18.6k
Forks
1.8k
Avg merge
1d 18h
Merged PRs (30d)
126

Description

In yesterday's Wasmtime meeting we had a discussion about testing, CI, and the artifacts that we produce. Currently the actual released artifacts are not actually tested in CI at all, they're simply produced. This isn't a great situation and ideally we would test the exact artifacts that CI produces rather than a different build configuration.

There's a few challenges to tackle when handling this though:

* Most of the test suite is not architected to run the `wasmtime` CLI or the `libwasmtime.so` C API. Some of `tests/all/cli_tests.rs` runs the `wasmtime` executable, but the executable is produced as part of `cargo test` and isn't a released artifact. The C API is only tested through the C examples (so very lightly).
* Testing artifacts might require a different command to execute or new infrastructure other than `cargo test`. It's unclear what this infrastructure would look like.
* Testing "all possible configurations" isn't feasible, even with the generous concurrency that we already have. For example we can't test the matrix of {platform, opt/debug, debug-asserts/no, crate} in CI as that would be too much.

We didn't reach many firm conclusions in the meeting but I wanted to open a follow-up issue to track further progress. A few points we did identify were:

* We should probably have one CI job test with `--release` to test with optimizations (just one platform though, probably not all of them)
* We should ideally run at least a "smoke test" for released artifacts.

cc @tschneidereit

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.