Add centralized tolerance registry for backend test harness
Open
@GregoryComer is already working on this.
Since Jun 1, 2026.
triaged
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
Centralized tolerance and capability registries for backend test harness
Summary
ExecuTorch's numerical tolerances and backend capability declarations are currently spread across individual test files and flow configs. This issue tracks centralizing both into backends/test/harness/.
This work focuses on the element-wise test infrastructure layer. Dataset-level accuracy tracking (e.g. top-1/top-5 over real datasets, as discussed in #13347) is a valuable but separate effort.
Problem
- Each backend defines its own tolerance defaults (ranging from
atol=1e-8toatol=1e-1) with 8+ different comparison mechanisms - No unified way to query what a backend supports (dtypes, dynamic shapes, quantization)
- Skip logic is spread across individual flow files with inconsistent patterns
Implementation
Tolerance registry
-
ToleranceConfigdataclass,BACKEND_TOLERANCESregistry (15 backends),get_tolerance()lookup API - Integration with shared
Testerharness — optionalbackendparam, fully backward-compatible - 37 unit tests
PR #20014 is open and under review.
Capability registry
-
BackendCapabilitiesdataclass,BACKEND_CAPABILITIESregistry (16 backends),should_skip_test()API - Integration with
TestFlow.should_skip_test()in the cross-backend suite - Migrate Vulkan dtype skip patterns and ARM dynamic shapes skip to registry
Tracked in #20015. PR to follow once #20014 lands.
Related Issues
- #13347 — Umbrella discussion on delegate testing strategy
- #20015 — Capability registry
- #18424 — ATOL/RTOL configurable per-test for Cortex-M BundleIO
- #14023 — ExecuTorch Test Infrastructure umbrella
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.