apache / apache/tvm

[Bug] tensordot produce inconsistency inference results when execuing twice under the same inputs

Open
#17,949 0 comments 0 reactions 0 assignees View on GitHub
needs-triage type: bug
Dominant language
Python
Stars
13.7k
Forks
4k
Avg merge
2d 19h
Merged PRs (30d)
111

Description

`topi.tensordot` cannot output a determined results under the same inputs. It's weird.

### Actual behavior

```
AssertionError:
Not equal to tolerance rtol=0.001, atol=0.001
An Inconsistency bug detected.
Mismatched elements: 1 / 1 (100%)
Max absolute difference among violations: 3.686663e+32
Max relative difference among violations: 1.
ACTUAL: array([-1.109184], dtype=float32)
DESIRED: array([3.686663e+32], dtype=float32)
```

### Environment

tvm-0.21.dev0

### Steps to reproduce

```
import tvm
from tvm import te, topi, tir
from tvm import meta_schedule as ms
import numpy as np

def compile_mod(mod, np_input_list, output_shape, output_type, opt_level=3):
with tvm.transform.PassContext(opt_level):
ref_mod = tvm.build(mod, target='llvm')
mod_output = tvm.nd.empty(output_shape, dtype=output_type, device=tvm.cpu(0))

tvm_inputs = [tvm.nd.array(x) for x in np_input_list]
ref_mod(*tvm_inputs, mod_output)
return mod_output

a = te.placeholder([1, 2, 3, 4], dtype='float32', name='a')
b = te.placeholder([4, 3, 2], dtype='float32', name='b')
op_output = topi.tensordot(a, b, axes=3)
np_inputs = [np.random.uniform(-1, 1, size=[1, 2, 3, 4]).astype('float32'),np.random.uniform(-1, 1, size=[4, 3, 2]).astype('float32')]
sch = tir.Schedule(te.create_prim_func([a, b, op_output]).with_attr('target', tvm.target.Target('llvm')))
output1 = compile_mod(sch.mod, np_inputs, op_output.shape, op_output.dtype, opt_level=3)
output2 = compile_mod(sch.mod, np_inputs, op_output.shape, op_output.dtype, opt_level=3)

np.testing.assert_allclose(
output1.numpy(), output2.numpy(), rtol=1e-3, atol=1e-3, err_msg=f"An inconsistency error."
```

### Triage

* needs-triage
* topi

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the provided reproducer using topi.tensordot, te.create_prim_func, tir.Schedule, and the LLVM target, then compare the generated execution across the two compile_mod calls. Trace the generated module and runtime behavior to identify why identical inputs produce different outputs. Done means repeated execution returns matching results within the stated tolerances.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.