[lit] Potential flakiness in llvm/utils/lit/tests/verbosity.py
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
It seems that `verbosity.py` [asserts on the order](https://github.com/llvm/llvm-project/blob/d3f58f9efd3e91824bc496e6fac80bad6d0db8f2/llvm/utils/lit/tests/verbosity.py#L11) in which tests are processed:
```
# NO-ARGS: -- Testing: 5 tests, 1 workers --
# NO-ARGS-NEXT: FAIL: verbosity :: fail.txt (1 of 5)
# NO-ARGS-NEXT: PASS: verbosity :: pass.txt (2 of 5)
# NO-ARGS-NEXT: {{UN}}SUPPORTED: verbosity :: unsupported.txt (3 of 5)
# NO-ARGS-NEXT: {{X}}FAIL: verbosity :: xfail.txt (4 of 5)
# NO-ARGS-NEXT: XPASS: verbosity :: xpass.txt (5 of 5)
```
However, in [this buildbot run](https://lab.llvm.org/buildbot/#/builders/212/builds/3833) the test failed due to ordering, as `unsupported.txt` is `(4 of 5)` instead of `(3 of 5)`.
```
# executed command: FileCheck /home/buildbot-worker/bbroot/clang-riscv-rva23-zvl512b-2stage/stage2/utils/lit/tests/verbosity.py --check-prefix NO-ARGS
# .---command stderr------------
# | /home/buildbot-worker/bbroot/clang-riscv-rva23-zvl512b-2stage/stage2/utils/lit/tests/verbosity.py:14:17: error: NO-ARGS-NEXT: expected string not found in input
# | # NO-ARGS-NEXT: {{UN}}SUPPORTED: verbosity :: unsupported.txt (3 of 5)
# | ^
# | :3:37: note: scanning from here
# | PASS: verbosity :: pass.txt (2 of 5)
# | ^
# | :5:1: note: possible intended match here
# | UNSUPPORTED: verbosity :: unsupported.txt (4 of 5)
```
Assuming that it's okay that these are done in a different order, should numbering assertions be removed, and maybe replacing the `NEXT` checks with `DAG` checks?
Contributor guide
Research direction
Start with llvm/utils/lit/tests/verbosity.py and the NO-ARGS checks around the reported line, then reproduce the failure with the FileCheck command shown in the issue or the corresponding lit test. Confirm whether result ordering is intentionally unspecified; done means the test accepts valid ordering while still checking all five results and passes on the affected build.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- compilers, testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100