Add backend capability registry for test harness
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
Add backend capability registry for test harness
Summary
Add a backend capability registry to enable capability-driven test selection across ExecuTorch backends. Complements the tolerance registry in #19910.
Problem
- No unified way to query what a backend supports (dtypes, dynamic shapes, quantization)
- Skip logic is spread across individual flow files (
skip_patterns,param_skip_reasons) with inconsistent patterns - Adding a new dtype or capability check means editing each backend's flow separately
Proposal
Add backends/test/harness/capabilities.py:
BackendCapabilitiesdataclass (supported_dtypes,supports_dynamic_shapes,supports_quantization,max_tensor_rank)BACKEND_CAPABILITIESregistry covering 16 backendsshould_skip_test(backend, dtype, ...)API returning a skip reason orNone- Integration into
TestFlow.should_skip_test()so tests auto-skip with consistent messages
Scope
This will:
- Add the registry and lookup API
- Integrate with
TestFlow.should_skip_test() - Migrate Vulkan dtype skips and ARM dynamic shapes skip to registry
Out of scope:
- Per-operator capability tracking (stays in backend partitioners)
- CI artifact generation
Depends on #20014 (tolerance registry) landing first since they share harness/__init__.py.
Related Issues
- #19910 — Tracking issue
- #13347 — Umbrella discussion on delegate testing strategy
cc: @GregoryComer
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading backends/test/harness/capabilities.py and the existing TestFlow.should_skip_test() implementation, along with the tolerance registry dependency in #20014. Define the registry and lookup API for the listed capabilities, integrate it into TestFlow, and migrate the Vulkan dtype and ARM dynamic-shape skips while covering the 16 backends described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100