test_Conv1d_zero_batch_xpu_fp32 fails under PYTORCH_TEST_WITH_DYNAMO=1: TypeError: add(): argument 'input' (position 1) must be Tensor, not NoneType
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 112
Description
### 🐛 Describe the bug
`TestNN.test_Conv1d_zero_batch_xpu_fp32` fails under `PYTORCH_TEST_WITH_DYNAMO=1` with a `TypeError` raised from the compiled-autograd traced backward graph. The failure reproduces on CPU-only code paths (no XPU kernel is ever invoked) and is triggered by test names introduced in https://github.com/pytorch/pytorch/pull/189653, which generalized `test/test_nn.py`'s `add_test()` to generate device-suffixed test names (e.g. `_xpu`) for any accelerator, not just `_cuda`. This exposed a pre-existing compiled-autograd gap that previously only affected CPU-named tests already present in `test/compiled_autograd_skips/` (e.g. `TestNN.test_Conv1d_circular_stride2_pad2`) but was never covered for the `_zero_batch` variant, and the new `_xpu` name has no skip entry at all.
### Reproduction steps
```bash
# Apply/checkout https://github.com/pytorch/pytorch/pull/189653 (or any revision after it lands)
# on an XPU-enabled PyTorch build.
PYTORCH_TEST_WITH_DYNAMO=1 python test/test_nn.py TestNN.test_Conv1d_zero_batch_xpu_fp32
```
### Mini-reproducer (no test harness needed)
This standalone script reproduces the crash purely on CPU, with no dependency on `test/test_nn.py` or `common_nn.py`. Save it as `repro_conv1d_zero_batch.py` and run `python repro_conv1d_zero_batch.py`:
```python
import torch
import torch.nn as nn
import torch._dynamo
torch._dynamo.config.compiled_autograd = True
def test_fn():
torch.manual_seed(0)
m = nn.Conv1d(4, 5, 3)
x = torch.randn(0, 4, 10, requires_grad=True) # zero batch dimension
out = m(x)
print("forward output shape:", out.shape)
grad_out = torch.randn_like(out, requires_grad=True)
# First-order grad with create_graph=True to enable a subsequent
# double-backward, mirroring NewModuleTest.test_device() in
# torch/testing/_internal/common_nn.py
gi, gw, gb = torch.autograd.grad(out, (x, m.weight, m.bias), grad_out, create_graph=True)
print("first-order grad OK:", gi.shape, gw.shape, gb.shape)
# Mix output into the second backward computation, exactly as
# test_device() does, "so that torch.autograd.grad doesn't complain
# that some inputs are unreachable"
outputs = out.sum() + gi.sum() + gw.sum() + gb.sum()
print("computing double-backward (this is expected to crash)...")
gg = torch.autograd.grad(
outputs,
(x, grad_out, m.weight, m.bias),
retain_graph=True,
) # <-- TypeError: add(): argument 'input' (position 1) must be Tensor, not NoneType
print("double-backward grads:", [g.shape if g is not None else None for g in gg])
# The torch._dynamo.optimize(...) wrapper mirrors how
# torch.testing._internal.common_utils.TestCase._run_custom() wraps the
# entire test method under PYTORCH_TEST_WITH_DYNAMO=1, which is what
# actually triggers the crash (a bare torch._dynamo.config.compiled_autograd=True
# without wrapping the whole call stack in dynamo does not reproduce it).
wrapped = torch._dynamo.optimize("eager_noexcept", nopython=False)(test_fn)
wrapped()
```
This confirms eager autograd silently zero-fills an undefined gradient contribution while compiled autograd's traced FX graph does not.
### Error log
```text
Traceback (most recent call last):
File ".../torch/fx/graph_module.py", line 495, in __call__
return super(self.cls, obj).__call__(*args, **kwargs)
File ".../torch/nn/modules/module.py", line 1780, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File ".../torch/nn/modules/module.py", line 1791, in _call_impl
return forward_call(*args, **kwargs)
File ".22", line 66, in forward
add = torch.add(getitem_31, getitem_38); getitem_31 = getitem_38 = None
TypeError: add(): argument 'input' (position 1) must be Tensor, not NoneType
Call using an FX-traced Module, line 66 of the traced Module's generated forward function:
getitem_40 = validate_outputs_10[2]; validate_outputs_10 = None
add = torch.add(getitem_31, getitem_38); getitem_31 = getitem_38 = None
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
add_1 = torch.add(getitem_32, getitem_39); getitem_32 = getitem_39 = None
```
Full pytest traceback:
```text
test/test_nn.py:7310: in with_tf32_off
test.test_device(self, **kwargs)
torch/testing/_internal/common_nn.py:3633: in test_device
cpu_gg = torch.autograd.grad(
outputs_cpu,
cpu_input_tuple + (cpu_gradOutput,) + tuple(cpu_module.parameters()),
retain_graph=True)
torch/autograd/__init__.py:594: in grad
result = _engine_run_backward(...)
torch/_dynamo/compiled_autograd.py:1273: in runtime_wrapper
out = compiled_fn(...)
...
torch/fx/graph_module.py:507: in __call__
raise e.with_traceback(None)
TypeError: add(): argument 'input' (position 1) must be Tensor, not NoneType
```
### Root cause analysis
1. `PYTORCH_TEST_WITH_DYNAMO=1` sets `torch._dynamo.config.compiled_autograd = True` globally for the test suite (`torch/testing/_internal/common_utils.py`, around line 1990).
2. `NewModuleTest.test_device()` (`torch/testing/_internal/common_nn.py:3554`) performs a double-backward ("gradgrad") check on a `Conv1d` module constructed with `input_size=(0, 4, 10)` (zero batch dimension) — see the `zero_batch` test entry at `common_nn.py:1206-1215`.
3. Because the batch dimension is zero, `torch.autograd.grad(..., create_graph=True)` produces a gradient w.r.t. the conv input that is fed through **two accumulating paths** that merge at the same `AccumulateGrad` node:
- one from `ConvolutionBackwardBackward0` (the double-backward of the conv operator itself)
- one from a second `ConvolutionBackward0` (from differentiating the `+ x.sum()` "mixing" term that `test_device()` adds specifically so `torch.autograd.grad` doesn't complain about unreachable inputs)
4. Compiled autograd traces both paths into an FX graph and inserts a `torch.add(getitem_31, getitem_38)` node to sum the two gradient contributions before final accumulation.
5. For the zero-batch case, one of `ConvolutionBackwardBackward0` / `ConvolutionBackward0`'s outputs corresponding to this gradient is **undefined (`None`)** instead of a correctly-shaped zero-sized tensor.
6. **In eager mode**, PyTorch's autograd engine silently substitutes a zero tensor for undefined outputs when accumulating multiple gradient contributions at a node with multiple incoming edges. **Compiled autograd's traced FX graph has no equivalent fallback** — it directly calls `torch.add(None, tensor)`, which is not a valid call and raises `TypeError`.
This is a real gap in either:
- the double-backward formula for convolution (`ConvolutionBackwardBackward`/`ConvolutionBackward`) not returning a correctly zero-shaped tensor for a zero-batch gradient, and/or
- compiled autograd's codegen for multi-edge gradient accumulation not being defensive against `None` the way the eager engine is.
Confirmed to be **CPU-only** and **not XPU-specific** — the crash occurs while computing `cpu_gg` in `test_device()`, before the GPU/XPU code path is ever reached (verified via `TORCH_LOGS="compiled_autograd_verbose"` trace, which shows the failing graph is `CompiledAutograd6` operating on `device(type='cpu')` tensors).
### Why this wasn't previously caught
`test/compiled_autograd_skips/` already contains skip entries for several structurally similar Conv double-backward gaps under compiled autograd (e.g. `TestNN.test_Conv1d_circular_stride2_pad2`, `TestNN.test_Conv2d_reflect_stride2_pad2`, `TestNN.test_Conv3d_replicate_stride2_pad2`), but **no entry exists for `TestNN.test_Conv1d_zero_batch`** (nor the `Conv2d`/`Conv3d` zero-batch siblings). This appears to be an omission from when those skip lists were originally generated via `scripts/compile_tests/update_failures.py`.
Additionally, prior to #189653, `add_test()` in `test/test_nn.py` only generated `_cuda`-suffixed test names for `test_cuda()`; there was no generic `_xpu` (or other accelerator) variant. #189653 renamed `test_cuda` → `test_device` and generalized the suffix to the current accelerator type (`torch.accelerator.current_accelerator()`), which means on an XPU-enabled build this test now surfaces as `test_Conv1d_zero_batch_xpu_fp32` — a name that was never covered by the (CPU-oriented) skip-list generation process.
### Suggested fix
1. **Immediate/minimal**: add `TestNN.test_Conv1d_zero_batch` (test-key is `ClassName.MethodName`, dtype suffixes like `_fp32`/`_tf32` are not part of the key used by `_dynamo_test_key()`) to `test/compiled_autograd_skips/` in pytorch/pytorch, following the existing convention (empty marker file). Consider doing the same for the `Conv2d_zero_batch` / `Conv3d_zero_batch` siblings if they exhibit the same failure.
2. **Root fix**: investigate why `ConvolutionBackwardBackward0` (or `ConvolutionBackward0`) returns an undefined/`None` gradient instead of a correctly zero-shaped tensor for a zero-batch input in the double-backward path, and/or make compiled autograd's multi-edge gradient-accumulation codegen tolerant of `None` inputs (mirroring the eager engine's behavior) rather than emitting a bare `torch.add(None, tensor)`.
3. Longer term: since #189653 will surface many more `_xpu`-suffixed test names that were never exercised by CPU-only skip-list generation, re-running `scripts/compile_tests/update_failures.py` on an XPU-enabled worker (or auditing `test/compiled_autograd_skips/` for missing device-suffixed siblings of already-skipped tests) would help catch the rest of this class of gap before it lands.
### Versions
```text
PyTorch version: 2.14.0.dev20260715+xpu
Is XPU available: True
XPU used to build PyTorch: 20260000
Intel GPU models onboard: Intel(R) Data Center GPU Max 1550 (x2)
OS: Ubuntu 24.04.4 LTS (x86_64)
Python version: 3.10.20
Reproduced with: PYTORCH_TEST_WITH_DYNAMO=1, applying pytorch/pytorch#189653
```
Contributor guide
Research direction
Start with test/compiled_autograd_skips/ and compare its existing Conv double-backward entries with the TestNN.test_Conv1d_zero_batch key described in the issue. Run PYTORCH_TEST_WITH_DYNAMO=1 python test/test_nn.py TestNN.test_Conv1d_zero_batch_xpu_fp32, then verify the relevant compiled-autograd test no longer fails or is reported without an appropriate skip; use torch/testing/_internal/common_nn.py and torch/_dynamo/compiled_autograd.py for deeper investigation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- compilers, testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100