[BMG] MixOrderReductionTest.test_rms_norm_bwd_with_dynamic_shape fails: RuntimeError: tensor does not have a device (eager rms_norm backward w/ dynamic shapes)
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 107
Description
## Description
Three parametrized cases of `MixOrderReductionTest.test_rms_norm_bwd_with_dynamic_shape` fail on XPU with `RuntimeError: tensor does not have a device`:
- `test_rms_norm_bwd_with_dynamic_shape_dynamic_dims0`
- `test_rms_norm_bwd_with_dynamic_shape_dynamic_dims1`
- `test_rms_norm_bwd_with_dynamic_shape_dynamic_dims2`
File: `test/inductor/test_mix_order_reduction.py` (class `MixOrderReductionTest`, base method `test_rms_norm_bwd_with_dynamic_shape`, decorated `@parametrize("dynamic_dims", ([0], [1], [0, 1]))`).
## Root cause
The failure occurs in the **eager reference backward pass**, before `torch.compile` runs. In the test, `ref = fwd_bwd(f)` (line 464) calls `out.backward(dy)` (line 445) on an eager `F.rms_norm` output where the input `x` was marked dynamic on dims (0, 1) via `torch._dynamo.mark_dynamic(x, (0, 1))`. The error is raised from the C++ autograd engine (`run_backward`), independent of the inductor mix-order-reduction codegen the test is meant to exercise.
This is an XPU backend / autograd bug, not a test-code issue: the test uses `device=GPU_TYPE` (device-agnostic), carries no `@skipIfXpu`, and the base method is present upstream (introduced by commit `93fef4bd1dd`, an ancestor of the tested HEAD).
## Reproduction
```
python -m pytest "test/inductor/test_mix_order_reduction.py::MixOrderReductionTest::test_rms_norm_bwd_with_dynamic_shape_dynamic_dims0" -v
```
Reproduced locally on `torch 2.13.0+xpu`:
```
FAILED [0.9454s] MixOrderReductionTest.test_rms_norm_bwd_with_dynamic_shape_dynamic_dims0
Traceback (most recent call last):
File "test/inductor/test_mix_order_reduction.py", line 464, in test_rms_norm_bwd_with_dynamic_shape
ref = fwd_bwd(f)
File "test/inductor/test_mix_order_reduction.py", line 445, in fwd_bwd
out.backward(dy)
File ".../torch/_tensor.py", line 623, in backward
torch.autograd.backward(...)
File ".../torch/autograd/__init__.py", line 395, in backward
_engine_run_backward(...)
File ".../torch/autograd/graph.py", line 979, in _engine_run_backward
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: tensor does not have a device
```
Full C++ origin (from the CI capture): `Exception raised from device_default at c10/core/TensorImpl.h:1302`.
## Environment
- torch: `2.13.0+xpu`
- Device: Intel GPU (XPU)
- Test config: `triton.mix_order_reduction` default-enabled (`True`)
## Additional context
Similar RMS-norm-backward tests in `MixOrderReductionTest` have prior (closed) DISABLED trackers upstream (pytorch/pytorch#177054, #177004, #176970), but none matches this specific method + XPU device error.
Contributor guide
Research direction
Run the listed pytest case in test/inductor/test_mix_order_reduction.py and inspect MixOrderReductionTest.test_rms_norm_bwd_with_dynamic_shape, especially fwd_bwd and the eager out.backward(dy) call. Trace the XPU autograd failure at device_default before torch.compile runs. Done means the three dynamic_dims cases complete on XPU while retaining coverage of the intended inductor path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100