llvm / llvm/llvm-project

Remove or enforce test uniqueness requirement in the LLDB API test suite

Open
#170,178 4 comments 0 reactions 0 assignees View on GitHub
lldb test-suite
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

Currently, the LLDB API test suite requires that test names (i.e. the name of the Python test class) are unique. However, this requirement only gets enforced when using `lldb-dotest` as the test runner (i.e. not `llvm-lit`). Some LLDB developers prefer using `lldb-dotest` during development for a variety of (good) reasons. However, `llvm-lit` is the canonical test runner that's used in CI and when using the `check-lldb*` targets, which is why we don't discover these name collisions sooner.

Naming collisions in tests pop up periodically and prompts the question why this keeps happening. There are two ways we can improve the current situation:

1. Move the check that catches this and bail out early. This way we uncover naming collisions as soon as they're introduced.
2. Drop the requirement. Based on Pavel's comment in https://github.com/llvm/llvm-project/pull/137681 it seems like the reason for its existence is no longer relevant.

We didn't reach consensus in https://github.com/llvm/llvm-project/pull/137681 so let's use this issue to settle on a direction.

Contributor guide

Open the contributing guide

Research direction

Review the LLDB API test suite and compare how lldb-dotest and llvm-lit handle duplicate Python test class names. Read the discussion in pull request 137681 first, then determine which behavior the project should adopt; the work is done when the chosen uniqueness policy is consistently enforced or removed and the relevant test-running paths agree.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers, devtools, testing-qa
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.