intel / intel/torch-xpu-ops

[BMG] MixOrderReductionTest.test_rms_norm_bwd_with_dynamic_shape fails: RuntimeError: tensor does not have a device (eager rms_norm backward w/ dynamic shapes)

Open
#4,233 0 comments 0 reactions 0 assignees View on GitHub
bug module: inductor test: ut
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 13h
Merged PRs (30d)
107

Description

## Description

Three parametrized cases of `MixOrderReductionTest.test_rms_norm_bwd_with_dynamic_shape` fail on XPU with `RuntimeError: tensor does not have a device`:

- `test_rms_norm_bwd_with_dynamic_shape_dynamic_dims0`
- `test_rms_norm_bwd_with_dynamic_shape_dynamic_dims1`
- `test_rms_norm_bwd_with_dynamic_shape_dynamic_dims2`

File: `test/inductor/test_mix_order_reduction.py` (class `MixOrderReductionTest`, base method `test_rms_norm_bwd_with_dynamic_shape`, decorated `@parametrize("dynamic_dims", ([0], [1], [0, 1]))`).

## Root cause

The failure occurs in the **eager reference backward pass**, before `torch.compile` runs. In the test, `ref = fwd_bwd(f)` (line 464) calls `out.backward(dy)` (line 445) on an eager `F.rms_norm` output where the input `x` was marked dynamic on dims (0, 1) via `torch._dynamo.mark_dynamic(x, (0, 1))`. The error is raised from the C++ autograd engine (`run_backward`), independent of the inductor mix-order-reduction codegen the test is meant to exercise.

This is an XPU backend / autograd bug, not a test-code issue: the test uses `device=GPU_TYPE` (device-agnostic), carries no `@skipIfXpu`, and the base method is present upstream (introduced by commit `93fef4bd1dd`, an ancestor of the tested HEAD).

## Reproduction

```
python -m pytest "test/inductor/test_mix_order_reduction.py::MixOrderReductionTest::test_rms_norm_bwd_with_dynamic_shape_dynamic_dims0" -v
```

Reproduced locally on `torch 2.13.0+xpu`:

```
FAILED [0.9454s] MixOrderReductionTest.test_rms_norm_bwd_with_dynamic_shape_dynamic_dims0
Traceback (most recent call last):
File "test/inductor/test_mix_order_reduction.py", line 464, in test_rms_norm_bwd_with_dynamic_shape
ref = fwd_bwd(f)
File "test/inductor/test_mix_order_reduction.py", line 445, in fwd_bwd
out.backward(dy)
File ".../torch/_tensor.py", line 623, in backward
torch.autograd.backward(...)
File ".../torch/autograd/__init__.py", line 395, in backward
_engine_run_backward(...)
File ".../torch/autograd/graph.py", line 979, in _engine_run_backward
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: tensor does not have a device
```

Full C++ origin (from the CI capture): `Exception raised from device_default at c10/core/TensorImpl.h:1302`.

## Environment

- torch: `2.13.0+xpu`
- Device: Intel GPU (XPU)
- Test config: `triton.mix_order_reduction` default-enabled (`True`)

## Additional context

Similar RMS-norm-backward tests in `MixOrderReductionTest` have prior (closed) DISABLED trackers upstream (pytorch/pytorch#177054, #177004, #176970), but none matches this specific method + XPU device error.

Contributor guide

Open the contributing guide

Research direction

Run the listed pytest case in test/inductor/test_mix_order_reduction.py and inspect MixOrderReductionTest.test_rms_norm_bwd_with_dynamic_shape, especially fwd_bwd and the eager out.backward(dy) call. Trace the XPU autograd failure at device_default before torch.compile runs. Done means the three dynamic_dims cases complete on XPU while retaining coverage of the intended inductor path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.