[Bug][CUDA][dlight] GPU Fallback calls sch.fuse on a zero loop extent and aborts with "Divide by zero"
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 4k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 112
Description
## Environment
- TVM version: `main @ 48242ec33403f2b6e4fac6e763ca7a683fb9d5df` (2026-09-03, reports `0.26.dev0`); the same trigger fails on `0.26.0`, and on `0.19.0` / `0.12.0` it surfaced as a `ZeroDivisionError` in Relay's `pad_to_tensorcore`.
- Build / install: source build, `USE_CUDA=ON`, nvcc 12.2, LLVM 15
- OS / Python: Ubuntu 24.04.3 LTS, x86_64, Python 3.10
- GPU / driver: Tesla V100-DGXS-32GB, driver 535.309.01, sm_70
- Target: `cuda`. **`opt_level` 3 and 0 both fail.** `llvm` is clean at both levels.
- **Pipeline coverage** (`relax.get_pipeline("default_build")` performs **no** operator fusion, so all three are reported separately). Verified 2026-09-06 on `0.26.0`, CUDA target — the abort happens at **compile** time, before any kernel launch:
- `get_pipeline("default_build")` (no FuseOps/FuseTIR) — **fails**
- `get_pipeline("zero")` (fuses) — **fails**
- forced `FuseOps` + `FuseTIR` — **fails**
(also confirmed on `main @ 48242ec` at `opt_level` 3 and 0)
## Minimal reproducer
Single `Neg` node with input `X : float32[4,0]` — a zero extent that is **not** on the leading axis. `model.onnx` + `feed.npz` + `run.py` attached.
```python
import numpy as np, onnx, tvm
from tvm import relax
from tvm.relax.frontend.onnx import from_onnx
model = onnx.load("model.onnx"); feed = dict(np.load("feed.npz")) # X: float32[4,0]
mod = from_onnx(model, shape_dict={k: list(v.shape) for k, v in feed.items()}, keep_params_in_input=False)
with tvm.transform.PassContext(opt_level=3):
tvm.compile(relax.get_pipeline("default_build")(mod), target="cuda") # <-- raises here
```
Companion note: the **leading**-axis zero-extent case does *not* fail here — it compiles and then faults at run time. The full matrix is: `0` on the leading axis (or a 1-D empty tensor) → runtime `cudaErrorIllegalAddress`; `0` on a non-leading axis → this compile-time abort.
## Expected vs actual
- **Expected**: compilation succeeds and the result is an empty `float32` tensor of shape `(4,0)`. `numpy`, onnxruntime, and TVM's own `llvm` target all produce that.
- **Actual**:
```
InternalError: Check failed: pb->value != 0 (0 vs. 0) : Divide by zero
```
raised during `tvm.compile`, at both `opt_level=3` and `opt_level=0`.
## Root cause (if known)
dlight's GPU `Fallback` rule calls `sch.fuse`, and `Fuse` reconstructs the index of a fused loop as `floordiv(floormod(fused, next_lower), lower)`. Here `next_lower == 0`, because it is the extent of the fused loop, which is 0. The zero is therefore **the compiler's own loop extent, not a divisor coming from the model** — an internal invariant that the scheduling rule breaks on itself. `llvm` is clean, so the rule is the producer.
Never fixed. On 0.12/0.19 the same trigger hit `pad_to_tensorcore` in the Relay pipeline (`ZeroDivisionError`); the site moved in `(0.19, main]`, the root cause did not.
## Why this is a bug (not tolerance / not undefined behaviour)
A zero extent on a non-leading axis is legal in ONNX and defined by `numpy`; TVM's `llvm` target handles it. The abort is not a validation message about the user's model — it is an `ICHECK` inside a scheduling rule about an expression the rule itself constructed.
**Related, not duplicate.** Open issue **#19922** and closed **#17745** hit the *same* `ICHECK` (`pb->value != 0`), but from `has_reshape_pattern` in the Relax analysis on the **llvm** target. That producer was fixed by merged PR **#19958** (2026-07-09), and our `llvm` controls are clean on `main @ 48242ec`. Different producer, different target, still live. Closed **#17938** is the same `ICHECK` string from a third producer (constant folding of an ONNX model).
## How found
Found by EquiAutomaton (equivalence-graph differential testing, TVM CUDA vs TVM llvm vs onnxruntime). Depth-0 — a single `Neg` node.
## Reproducer archive
[TVM-C3-reproducer.zip](https://github.com/user-attachments/files/31877188/TVM-C3-reproducer.zip)
## Triage
- Needs triage
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the attached run.py reproducer and trace dlight's GPU Fallback rule through sch.fuse and Fuse, where the zero loop extent is reconstructed. Compare the CUDA compilation path with the clean LLVM control. Done means the non-leading-axis empty tensor compiles successfully and produces the expected float32 shape (4,0) result without the divide-by-zero abort.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- compilers, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100