Reduce reliance on E2E Tests
- Dominant language
- MLIR
- Stars
- 906
- Forks
- 171
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 32
Description
During the office hour today, we discussed a few ways to improve the developer experience around the slow build/test (a fresh build takes around 20 min on the most recent top-of-the-line MacBook Air, and can take easly twice/three times as long on older hardware), and Cindy suggested we split our long-running tests off into a separate group (that'd be mostly the E2E test), with the usual edit-compile-test loop using only the faster running tests.
I think this is a great idea, but we currently rely on the E2E tests a lot, as they're really the only thing that tests that IR output by one pass matches the expectations of the passes/pipeline. In order for the "quick" test set to be a more useful indicator, we need to improve test coverage for this in the normal Lit/FileCheck tests.
In the past, we already discussed that we want to establish clearer pre-/post-conditions for passes/pipelines (See #1985) and, ideally, also actively enforce these constraints with some kind of IR walk (this would also allow us to generate nicer error messages than just "could not convert illegal op .."). I think this would tie in really well with this issue, as we could use those pre/post-condition checkers along with pass-by-pass tests to better catch pipeline-violating changes w/o always needing to run full pipelines (and defintivelty not E2E w/ backends/etc)
Contributor guide
Assessment
This issue has not been assessed yet.