[Bug Skip]: test_activation_checkpointing_xpu.test_attr_compile_submodules_in_checkpoint_wrapper failed with untimeError( RuntimeError: PyTorch's device state was initialized in the forward pass of a Checkpoint, which is not allowed.
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 112
Description
### 🐛 Describe the bug with skip template
#### Observed behavior
RuntimeError: PyTorch's device state was initialized in the forward pass of a Checkpoint, which is not allowed.
The failure is raised from torch/utils/checkpoint.py in _checkpoint_without_reentrant_generator_impl when running:
return checkpoint(self.block, x, use_reentrant=False)
--------
### Skip
Cases:
op_ut,third_party.torch-xpu-ops.test.xpu.dynamo.test_activation_checkpointing_xpu.RematerializeACNodesPassTests,test_attr_compile_submodules_in_checkpoint_wrapper
--------
### Error Information
From traceback:
```
Test call site: test/xpu/dynamo/test_activation_checkpointing_xpu.py:3260
Forward call site: test/xpu/dynamo/test_activation_checkpointing_xpu.py:3242
Runtime error raised at: torch/utils/checkpoint.py:2005
```
failed test case
```
@torch._dynamo.config.patch(skip_fwd_side_effects_in_bwd_under_checkpoint=True)
def test_attr_compile_submodules_in_checkpoint_wrapper(self):
"""Compiling submodules inside a checkpointed block should not hit the
recompile limit due to WeakKeyDictionary guards in the pack_hook."""
from torch.utils.checkpoint import checkpoint
class Block(nn.Module):
def __init__(self, dim):
super().__init__()
self.norm1 = nn.RMSNorm(dim)
self.linear1 = nn.Linear(dim, dim, bias=False)
self.norm2 = nn.RMSNorm(dim)
self.linear2 = nn.Linear(dim, dim, bias=False)
self.norm3 = nn.RMSNorm(dim)
self.linear3 = nn.Linear(dim, dim, bias=False)
def forward(self, x):
x = x + self.linear1(self.norm1(x))
x = x + self.linear2(self.norm2(x))
x = x + self.linear3(self.norm3(x))
return x
class CheckpointedBlock(nn.Module):
def __init__(self, block):
super().__init__()
self.block = block
def forward(self, x):
return checkpoint(self.block, x, use_reentrant=False)
dim = 32
block = Block(dim)
x_ref = torch.randn(4, dim, requires_grad=True)
ref = block(x_ref)
ref.sum().backward()
block_cp = Block(dim)
block_cp.load_state_dict(block.state_dict())
wrapped = CheckpointedBlock(block_cp)
for _, submod in wrapped.block.named_children():
submod.compile(backend="aot_eager")
with torch._dynamo.config.patch(recompile_limit=2):
x_test = x_ref.detach().clone().requires_grad_(True)
result = wrapped(x_test)
result.sum().backward()
self.assertEqual(ref, result)
self.assertEqual(x_ref.grad, x_test.grad)
```
checkpoint.py
```
if forward_context_suppressed_exc:
raise RuntimeError(
"torch.utils.checkpoint: the forward context provided by "
"context_fn suppressed an exception raised during the "
"checkpointed forward. This is not supported because checkpoint "
"cannot return a value for a failed forward."
)
new_frame.forward_completed = True
if getattr(device_module, "_initialized", False) and \
preserve_rng_state and not had_device_in_fwd: # type: ignore[possibly-undefined]
# Device was not initialized before running the forward, so we didn't
# stash the device state.
raise RuntimeError(
"PyTorch's device state was initialized in the forward pass "
"of a Checkpoint, which is not allowed. Please open an issue "
"if you need this feature."
)
return
```
#### Expected behavior
The checkpointed forward pass should not trigger device-state initialization during checkpoint execution, and the test should complete without raising the above RuntimeError.
Reproduction
From the repository root:
PYTORCH_TEST_WITH_SLOW=1 python test/xpu/dynamo/test_activation_checkpointing_xpu.py RematerializeACNodesPassTests.test_attr_compile_submodules_in_checkpoint_wrapper
(Optionally suppress repro hint output in failures with PYTORCH_PRINT_REPRO_ON_FAILURE=0.)
### XPU PR
https://github.com/intel/torch-xpu-ops/pull/4894
### Versions
Version context from report: pytorch main 2.15.0a0
pytorch main 2.15.0a0
Contributor guide
Research direction
Run the named test from test/xpu/dynamo/test_activation_checkpointing_xpu.py:3260 using the reproduction command. Read the checkpoint handling in torch/utils/checkpoint.py around line 2005 and compare the behavior with the linked XPU PR. Done means the checkpointed forward and backward complete without the device-state RuntimeError and the equality assertions pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100