llvm / llvm/offload-test-suite

Unify RUN-line -T profile with SM_X_Y REQUIRES

Open
#1,219 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
18
Forks
39
Avg merge
2d 18h
Merged PRs (30d)
40

Description

PR #1195 added explicit # REQUIRES: SM_X_Y to several tests so that machines whose GPU does not support the shader model in the RUN line skip the tests instead of failing. See the original CI failure on a sub-SM_6_6 machine:
https://github.com/llvm/offload-test-suite/actions/runs/25931189782/job/76225188838#step:13:338

In the same review, @farzonl observed that the -T cs_X_Y profile in the RUN line and the SM_X_Y token in REQUIRES must be kept in sync manually, which is easy to get wrong over time.

This issue tracks unifying the two so the profile in the RUN line drives the SM gate automatically.

Proposed design:

Introduce a lit.formats.ShTest subclass in test/lit.cfg.py that, when it executes, scans the test file for -T _X_Y substrings, computes the maximum SM seen, and treats it as an implicit SM_X_Y REQUIRES. If the corresponding availability feature is not present, the test
returns UNSUPPORTED instead of failing.

Sketch:
```
import re
import lit.formats, lit.Test

class OffloadShTest(lit.formats.ShTest):
PROFILE_RE = re.compile(r"-T\s+\w{2,3}_(\d)_(\d)") # cs_6_6, ps_6_5, lib_6_3, ...

def execute(self, test, lit_cfg):
try:
content = open(test.getSourcePath()).read()
except OSError:
return super().execute(test, lit_cfg)
profiles = [(int(M), int(m)) for M, m in self.PROFILE_RE.findall(content)]
if profiles:
major, minor = max(profiles)
needed = f"SM_{major}_{minor}"
if needed not in test.config.available_features:
return lit.Test.Result(lit.Test.UNSUPPORTED,
f"derived requirement {needed} not met")
return super().execute(test, lit_cfg)

config.test_format = OffloadShTest()
```
Combined with the existing iterative SM_6_X availability features in lit.cfg.py, authors would no longer need to write # REQUIRES: SM_X_Y manually; bumping the -T profile would automatically bump the gate.

Follow-up work

- Implement the custom test format.
- Remove the now-redundant # REQUIRES: SM_X_Y tokens added in PR #1195 and elsewhere.
- Add a small test for the format itself (e.g., a fixture asserting UNSUPPORTED behavior on a synthetic high-SM test).

Trade-offs / caveats

- Regex parsing of RUN lines is fragile vs. lit substitutions. Tests that hide the profile behind a substitution (e.g. %dxc_default_target) would be invisible to the heuristic. Today our tests embed the profile literally, so this is fine, but it is a coupling point.
- Tests that intentionally need a higher gate than the RUN line implies (uncommon) would keep an explicit additional # REQUIRES:. The implicit gate is additive, not a replacement.
- Only the SM gate is unified. Other capability gates (Int64, VulkanInt64BufferAtomics, etc.) still need explicit REQUIRES.
- Adds a small amount of lit machinery for future maintainers to understand.

Issue creation assisted by: Github Copilot

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with test/lit.cfg.py and the tests containing the SM_X_Y requirements added by PR #1195. Implement and register the custom lit test format described in the issue, then add a focused fixture for an unavailable derived SM requirement. Done means literal -T profiles derive the maximum SM gate, unsupported tests are skipped, redundant requirements are removed, and existing tests still run.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.