apache / apache/tvm

[Bug][CUDA] A tensor with a leading zero extent compiles, and the generated kernel faults with cudaErrorIllegalAddress at the first StreamSync

Open
#20,278 0 comments 0 reactions 0 assignees View on GitHub
needs-triage type: bug
Dominant language
Python
Stars
13.7k
Forks
4k
Avg merge
2d 1h
Merged PRs (30d)
112

Description

## Environment
- TVM version: `main @ 48242ec33403f2b6e4fac6e763ca7a683fb9d5df` (2026-09-03 21:20:55 -0400, reports `0.26.dev0`); also present on release `0.26.0`, `0.19.0` and `0.12.0`.
- Build / install: source build (`/home/lxx/tvm-main`), `USE_CUDA=ON`, nvcc 12.2, LLVM 15
- OS / Python: Ubuntu 24.04.3 LTS, x86_64 (Xeon E5-2698 v4), Python 3.10
- GPU / driver: Tesla V100-DGXS-32GB, driver 535.309.01, `CUDA_VISIBLE_DEVICES=0`, sm_70
- Target: `cuda`. **`opt_level` 3 and 0 both fail.** `llvm` is clean.
- Every run is **one case per fresh process** — `cudaErrorIllegalAddress` is sticky for the life of a CUDA context.

- **Pipeline coverage** (`relax.get_pipeline("default_build")` performs **no** operator fusion, so all three are reported separately). Verified 2026-09-06 on `main @ 48242ec`, CUDA, **one pipeline per fresh process** (a `cudaErrorIllegalAddress` poisons the context for the life of the process):
- `get_pipeline("default_build")` (no FuseOps/FuseTIR) — **fails**, 2/2 fresh processes
- `get_pipeline("zero")` (fuses) — **fails**, fresh process
- forced `FuseOps` + `FuseTIR` appended to the legalize/fold sequence — **fails**, fresh process

## Minimal reproducer
Single `Neg` node with input `X : float32[0]`. `model.onnx` + `feed.npz` + `run.py` attached (a `MaxPool` variant, `model_maxpool_alt.onnx` + `feed_maxpool_alt.npz`, is included to show the trigger is the shape, not the operator).

```python
import numpy as np, onnx, tvm
from tvm import relax
from tvm.relax.frontend.onnx import from_onnx
model = onnx.load("model.onnx"); feed = dict(np.load("feed.npz")) # X: float32[0]
mod = from_onnx(model, shape_dict={k: list(v.shape) for k, v in feed.items()}, keep_params_in_input=False)
order = [i.name for i in model.graph.input]
with tvm.transform.PassContext(opt_level=3):
ex = tvm.compile(relax.get_pipeline("default_build")(mod), target="cuda") # succeeds
dev = tvm.cuda(0)
out = relax.VirtualMachine(ex, dev)["main"](*[tvm.runtime.tensor(np.ascontiguousarray(feed[n]), dev) for n in order])
print(np.asarray(out.numpy())) # <-- faults here
```

## Expected vs actual
- **Expected**: an empty `float32` tensor of shape `(0,)`. `numpy.negative(numpy.zeros((0,), numpy.float32))` is an empty array; onnxruntime returns the same; the identical program on TVM's `llvm` target returns the same.
- **Actual**: `tvm.compile` **succeeds** and the VM call **succeeds**; the failure surfaces at the first synchronisation:
```
CUDA Runtime Error: cudaErrorIllegalAddress (700)
cuda_device_api.cc:279 CUDADeviceAPI::StreamSync
<- runtime/tensor.cc:104 Tensor::CopyToBytes
```
After this the whole CUDA context is poisoned; every later allocation in the same process fails.
- Because the fault is asynchronous, a validator that never copies the empty result back to the host will report a **pass**. (This is why an earlier bisect of ours wrongly recorded a "(0.12, 0.19] regression" — the 0.12 run never triggered the sync. Corrected: the defect is present on 0.12, 0.19, 0.26.0 and main alike.)

## Root cause (if known)
The scheduled `PrimFunc` for shape `(0,)` carries **no thread binding at all** — the dlight/GPU schedule gives up on the zero-extent loop instead of binding it or guarding it. The launched kernel therefore addresses memory it does not own.

Relevant prior art in the tree: merged PR **#7273** deliberately guaranteed at least one block for empty tensors on the (then) GPU path. That guarantee does not hold on the current Relax/dlight CUDA pipeline.

`llvm` is clean at both opt levels, so the defect is in the CUDA scheduling path, not in `Neg`.

## Why this is a bug (not tolerance / not undefined behaviour)
A zero-extent tensor is legal — ONNX permits it, `numpy` defines every operation on it, and TVM's own `llvm` target handles it. Compiling to a kernel that performs an illegal memory access is a memory-safety defect: it corrupts the CUDA context of the whole process, and because the fault is asynchronous it can be attributed to whatever unrelated code happens to synchronise next. Even if one argued the empty case should be rejected, silently emitting a faulting kernel is not an acceptable rejection.

## How found
Found by EquiAutomaton (equivalence-graph differential testing against onnxruntime, TVM CUDA vs TVM llvm). Depth-0 — a single `Neg` node. 15 observations in the original 0.12 campaign; re-verified on `main @ 48242ec` in a fresh process, 2/2.

## Reproducer archive
[TVM-C2-reproducer.zip](https://github.com/user-attachments/files/31877189/TVM-C2-reproducer.zip)

## Triage
- Needs triage

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the minimal Neg reproducer in the issue and inspect the scheduled PrimFunc produced by relax.get_pipeline("default_build") for shape (0, comparing the CUDA and LLVM targets. Trace the dlight/GPU scheduling path and the prior empty-tensor behavior referenced by PR #7273. Done means the CUDA case returns an empty float32 tensor without an illegal address at synchronization across the listed pipelines.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.