[Relax][ONNX] Use ONNX Backend Tests to improve frontend coverage
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 4k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 112
Description
This is my brain dump for a possible improvement to the Relax ONNX frontend tests.
Currently, many ONNX frontend tests in Relax manually construct small ONNX models and compare the imported Relax program against ONNX Runtime. This makes the test suite somewhat ad-hoc. As coverage grows, it becomes easy to miss ONNX spec corner cases, opset-specific behavior, and edge cases that are already covered by the official ONNX test suite.
ONNX provides an official Backend Test suite, including node-level tests for operator behavior:
- https://github.com/onnx/onnx/blob/main/docs/OnnxBackendTest.md
I think it would be worth exploring whether Relax's ONNX frontend tests can reuse those tests as a more systematic source of coverage.
The goal is not necessarily to expose TVM as a full ONNX backend. Rather, the idea is to use the ONNX Backend Tests as a testing mechanism for the importer:
- improve coverage of ONNX operator semantics
- reduce duplicated hand-written operator tests
- catch edge cases that are easy to miss with random-input tests
- align frontend correctness testing more closely with the ONNX specification
- keep TVM-specific frontend tests only for behavior that ONNX Backend Tests do not cover, such as importer options, parameter handling, name sanitization, and Relax-specific integration
Contributions, suggestions, or alternative ideas would be very welcome.
This issue is intended to collect thoughts and discuss whether this direction makes sense before working out a detailed design.
cc @KJlaccHoeUM9l @junrushao
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the ONNX Backend Test documentation and the existing Relax ONNX frontend tests mentioned in the issue. Determine whether those tests can systematically cover importer behavior while preserving TVM-specific tests; done should include an agreed design and scope before implementation begins.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100