llvm / llvm/llvm-project

[lit] Add a way to prevent undesirably skipping tests

Open
#164,681 0 comments 0 reactions 0 assignees View on GitHub
llvm-lit
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

Lit uses "features" to enable and disable individual tests with e.g. `REQUIRES: floating-point-support` or `UNSUPPORTED: windows`. This is used to create often complex conditions that enable/disable parts of a test suite.

This is great, except that as these conditions grow in number and complexity, we can sometimes end up disabling parts of the test suite that shouldn't be disabled. When the test suite uses `UNSUPPORTED:` or `REQUIRES:`, tests can end up never running without anybody noticing.

It would be nice to have an easy way to detect when that happens. Here are a few possibilities:

1. Provide a way for `lit` to check the list of features defined in every test against a user-provided list of features. If the features don't match, make it an error. For example: `lit -sv libcxx/tests --check-features /path/to/list/of/features/txt`. That way, for example, libc++ pre-commit CI runners would pass the list of features they expect to `lit` and we'd see an error whenever that changes. It would require some maintenance but it might be manageable.
2. Provide a way for `lit` to check the list of skipped tests against a user-provided list. Same idea as (1), except we provide a list of tests instead of a list of features.
3. Alternative designs to explore might be to extend (or reuse) the support we already have for setting the list of tests to `XFAIL` via `--xfail LIST`. Perhaps we can do something similar like `--unsupported LIST`, however we'd have to think through the implications of controlling XFAILs/UNSUPPORTEDs solely via that mechanism.

Contributor guide

Open the contributing guide

Research direction

Start by reading lit's existing feature handling and the current --xfail LIST support, using the libcxx/tests example as the context. Compare the proposed feature-list, skipped-test-list, and --unsupported designs, then define which behavior and validation rules would make unexpected skips detectable.

Written by the indexing model from the issue text.

Assessment

Domain
cli, testing, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.