pytorch / pytorch/executorch

Add centralized tolerance registry for backend test harness

Open
#19,910 3 comments 0 reactions 1 assignee View on GitHub

@GregoryComer is already working on this.

Since Jun 1, 2026.

triaged
Dominant language
Python
Stars
5k
Forks
1.2k
Avg merge
2d 10h
Merged PRs (30d)
581

Description

Centralized tolerance and capability registries for backend test harness

Summary

ExecuTorch's numerical tolerances and backend capability declarations are currently spread across individual test files and flow configs. This issue tracks centralizing both into backends/test/harness/.

This work focuses on the element-wise test infrastructure layer. Dataset-level accuracy tracking (e.g. top-1/top-5 over real datasets, as discussed in #13347) is a valuable but separate effort.

Problem
  • Each backend defines its own tolerance defaults (ranging from atol=1e-8 to atol=1e-1) with 8+ different comparison mechanisms
  • No unified way to query what a backend supports (dtypes, dynamic shapes, quantization)
  • Skip logic is spread across individual flow files with inconsistent patterns
Implementation
Tolerance registry
  • ToleranceConfig dataclass, BACKEND_TOLERANCES registry (15 backends), get_tolerance() lookup API
  • Integration with shared Tester harness — optional backend param, fully backward-compatible
  • 37 unit tests

PR #20014 is open and under review.

Capability registry
  • BackendCapabilities dataclass, BACKEND_CAPABILITIES registry (16 backends), should_skip_test() API
  • Integration with TestFlow.should_skip_test() in the cross-backend suite
  • Migrate Vulkan dtype skip patterns and ARM dynamic shapes skip to registry

Tracked in #20015. PR to follow once #20014 lands.

Related Issues
  • #13347 — Umbrella discussion on delegate testing strategy
  • #20015 — Capability registry
  • #18424 — ATOL/RTOL configurable per-test for Cortex-M BundleIO
  • #14023 — ExecuTorch Test Infrastructure umbrella

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.