apache / apache/tvm

[Relax][ONNX] Use ONNX Backend Tests to improve frontend coverage

Open
#19,505 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
13.7k
Forks
4k
Avg merge
2d 1h
Merged PRs (30d)
112

Description

This is my brain dump for a possible improvement to the Relax ONNX frontend tests.

Currently, many ONNX frontend tests in Relax manually construct small ONNX models and compare the imported Relax program against ONNX Runtime. This makes the test suite somewhat ad-hoc. As coverage grows, it becomes easy to miss ONNX spec corner cases, opset-specific behavior, and edge cases that are already covered by the official ONNX test suite.

ONNX provides an official Backend Test suite, including node-level tests for operator behavior:
- https://github.com/onnx/onnx/blob/main/docs/OnnxBackendTest.md

I think it would be worth exploring whether Relax's ONNX frontend tests can reuse those tests as a more systematic source of coverage.

The goal is not necessarily to expose TVM as a full ONNX backend. Rather, the idea is to use the ONNX Backend Tests as a testing mechanism for the importer:

- improve coverage of ONNX operator semantics
- reduce duplicated hand-written operator tests
- catch edge cases that are easy to miss with random-input tests
- align frontend correctness testing more closely with the ONNX specification
- keep TVM-specific frontend tests only for behavior that ONNX Backend Tests do not cover, such as importer options, parameter handling, name sanitization, and Relax-specific integration

Contributions, suggestions, or alternative ideas would be very welcome.

This issue is intended to collect thoughts and discuss whether this direction makes sense before working out a detailed design.

cc @KJlaccHoeUM9l @junrushao

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the ONNX Backend Test documentation and the existing Relax ONNX frontend tests mentioned in the issue. Determine whether those tests can systematically cover importer behavior while preserving TVM-specific tests; done should include an agreed design and scope before implementation begins.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.