KhronosGroup / KhronosGroup/SPIRV-Tools

[RFC] Add a tool to automatically generate CHECK comments from input SPIR-V

Open
#4,839 3 comments 1 reaction 0 assignees View on GitHub
dev:testing feature request rfc
Dominant language
C++
Stars
1.4k
Forks
709
Avg merge
1d 22h
Merged PRs (30d)
28

Description

spirv-tools, as well as other open source projects like https://github.com/microsoft/DirectXShaderCompiler, use effcee for unit/regression testing. To create a test out of an expected SPIR-V file, one has to add `; CHECK: ` comments matching the instructions of interest and their dependencies ('inputs'). This is tedious and can be automated, so I'd like to propose a new tool to automatically generate effcee checks.

The LLVM project has a few scripts to generate and update `FileCheck` checks, e.g., https://github.com/llvm/llvm-project/blob/main/llvm/utils/update_test_checks.py. @dneto0 wrote a similar tool for clspv: https://github.com/google/clspv/blob/main/test/spvasm2checks.pl.

To make such a tool usable, we would want to limit the number of `CHECK` lines so that only the instructions of interest and their dependencies are matched (e.g., by following use-def chains). Similarly, the `CHECK`s should not care about the value naming or the exact order of types/constants.

The command line interface could look something like this:
```shell
./generate_checks --match="%foo,%bar,11,:15" [--use-strict-order] [--preserve-control-flow]
```

where `--match` specifies the list of instructions to match (by name, result ID, or line number (in disassembly)), `--use-strict-order` decides whether to use `CHECK` or `CHECK-DAG` for constants and types, and `--preserve-control-flow` decides whether to preserve all control flow constructs.

---

cc: @dneto0, @s-perron, @dnovillo

Contributor guide

Open the contributing guide

Research direction

Start by reading LLVM's update_test_checks.py and clspv's test/spvasm2checks.pl, then review the proposed command-line options and matching rules in this issue. Clarify the intended scope, dependency and ordering behavior, and supported input formats before implementation. Done means the project has an agreed design and a working generator with tests covering the specified behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
testing-qa, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.