google / google/heir

Reduce reliance on E2E Tests

Open
#2,084 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
MLIR
Stars
906
Forks
171
Avg merge
4d 12h
Merged PRs (30d)
32

Description

During the office hour today, we discussed a few ways to improve the developer experience around the slow build/test (a fresh build takes around 20 min on the most recent top-of-the-line MacBook Air, and can take easly twice/three times as long on older hardware), and Cindy suggested we split our long-running tests off into a separate group (that'd be mostly the E2E test), with the usual edit-compile-test loop using only the faster running tests.

I think this is a great idea, but we currently rely on the E2E tests a lot, as they're really the only thing that tests that IR output by one pass matches the expectations of the passes/pipeline. In order for the "quick" test set to be a more useful indicator, we need to improve test coverage for this in the normal Lit/FileCheck tests.

In the past, we already discussed that we want to establish clearer pre-/post-conditions for passes/pipelines (See #1985) and, ideally, also actively enforce these constraints with some kind of IR walk (this would also allow us to generate nicer error messages than just "could not convert illegal op .."). I think this would tie in really well with this issue, as we could use those pre/post-condition checkers along with pass-by-pass tests to better catch pipeline-violating changes w/o always needing to run full pipelines (and defintivelty not E2E w/ backends/etc)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.