llvm / llvm/llvm-project

[lit] Potential flakiness in llvm/utils/lit/tests/verbosity.py

Open Beginner friendly
#211,923 0 comments 0 reactions 0 assignees View on GitHub
tools:llvm-lit
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

It seems that `verbosity.py` [asserts on the order](https://github.com/llvm/llvm-project/blob/d3f58f9efd3e91824bc496e6fac80bad6d0db8f2/llvm/utils/lit/tests/verbosity.py#L11) in which tests are processed:

```
# NO-ARGS: -- Testing: 5 tests, 1 workers --
# NO-ARGS-NEXT: FAIL: verbosity :: fail.txt (1 of 5)
# NO-ARGS-NEXT: PASS: verbosity :: pass.txt (2 of 5)
# NO-ARGS-NEXT: {{UN}}SUPPORTED: verbosity :: unsupported.txt (3 of 5)
# NO-ARGS-NEXT: {{X}}FAIL: verbosity :: xfail.txt (4 of 5)
# NO-ARGS-NEXT: XPASS: verbosity :: xpass.txt (5 of 5)
```

However, in [this buildbot run](https://lab.llvm.org/buildbot/#/builders/212/builds/3833) the test failed due to ordering, as `unsupported.txt` is `(4 of 5)` instead of `(3 of 5)`.

```
# executed command: FileCheck /home/buildbot-worker/bbroot/clang-riscv-rva23-zvl512b-2stage/stage2/utils/lit/tests/verbosity.py --check-prefix NO-ARGS
# .---command stderr------------
# | /home/buildbot-worker/bbroot/clang-riscv-rva23-zvl512b-2stage/stage2/utils/lit/tests/verbosity.py:14:17: error: NO-ARGS-NEXT: expected string not found in input
# | # NO-ARGS-NEXT: {{UN}}SUPPORTED: verbosity :: unsupported.txt (3 of 5)
# | ^
# | :3:37: note: scanning from here
# | PASS: verbosity :: pass.txt (2 of 5)
# | ^
# | :5:1: note: possible intended match here
# | UNSUPPORTED: verbosity :: unsupported.txt (4 of 5)
```

Assuming that it's okay that these are done in a different order, should numbering assertions be removed, and maybe replacing the `NEXT` checks with `DAG` checks?

Contributor guide

Open the contributing guide

Research direction

Start with llvm/utils/lit/tests/verbosity.py and the NO-ARGS checks around the reported line, then reproduce the failure with the FileCheck command shown in the issue or the corresponding lit test. Confirm whether result ordering is intentionally unspecified; done means the test accepts valid ordering while still checking all five results and passes on the affected build.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers, testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.