llvm / llvm/offload-test-suite
Unify RUN-line -T profile with SM_X_Y REQUIRES
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 18
- Forks
- 39
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 40
Description
PR #1195 added explicit # REQUIRES: SM_X_Y to several tests so that machines whose GPU does not support the shader model in the RUN line skip the tests instead of failing. See the original CI failure on a sub-SM_6_6 machine:
https://github.com/llvm/offload-test-suite/actions/runs/25931189782/job/76225188838#step:13:338
In the same review, @farzonl observed that the -T cs_X_Y profile in the RUN line and the SM_X_Y token in REQUIRES must be kept in sync manually, which is easy to get wrong over time.
This issue tracks unifying the two so the profile in the RUN line drives the SM gate automatically.
Proposed design:
Introduce a lit.formats.ShTest subclass in test/lit.cfg.py that, when it executes, scans the test file for -T _X_Y substrings, computes the maximum SM seen, and treats it as an implicit SM_X_Y REQUIRES. If the corresponding availability feature is not present, the test
returns UNSUPPORTED instead of failing.
Sketch:
```
import re
import lit.formats, lit.Test
class OffloadShTest(lit.formats.ShTest):
PROFILE_RE = re.compile(r"-T\s+\w{2,3}_(\d)_(\d)") # cs_6_6, ps_6_5, lib_6_3, ...
def execute(self, test, lit_cfg):
try:
content = open(test.getSourcePath()).read()
except OSError:
return super().execute(test, lit_cfg)
profiles = [(int(M), int(m)) for M, m in self.PROFILE_RE.findall(content)]
if profiles:
major, minor = max(profiles)
needed = f"SM_{major}_{minor}"
if needed not in test.config.available_features:
return lit.Test.Result(lit.Test.UNSUPPORTED,
f"derived requirement {needed} not met")
return super().execute(test, lit_cfg)
config.test_format = OffloadShTest()
```
Combined with the existing iterative SM_6_X availability features in lit.cfg.py, authors would no longer need to write # REQUIRES: SM_X_Y manually; bumping the -T profile would automatically bump the gate.
Follow-up work
- Implement the custom test format.
- Remove the now-redundant # REQUIRES: SM_X_Y tokens added in PR #1195 and elsewhere.
- Add a small test for the format itself (e.g., a fixture asserting UNSUPPORTED behavior on a synthetic high-SM test).
Trade-offs / caveats
- Regex parsing of RUN lines is fragile vs. lit substitutions. Tests that hide the profile behind a substitution (e.g. %dxc_default_target) would be invisible to the heuristic. Today our tests embed the profile literally, so this is fine, but it is a coupling point.
- Tests that intentionally need a higher gate than the RUN line implies (uncommon) would keep an explicit additional # REQUIRES:. The implicit gate is additive, not a replacement.
- Only the SM gate is unified. Other capability gates (Int64, VulkanInt64BufferAtomics, etc.) still need explicit REQUIRES.
- Adds a small amount of lit machinery for future maintainers to understand.
Issue creation assisted by: Github Copilot
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with test/lit.cfg.py and the tests containing the SM_X_Y requirements added by PR #1195. Implement and register the custom lit test format described in the issue, then add a focused fixture for an unavailable derived SM requirement. Done means literal -T profiles derive the maximum SM gate, unsupported tests are skipped, redundant requirements are removed, and existing tests still run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100