torch.compile error when creating nested tensors dynamically in forward pass
- Dominant language
- Python
- Stars
- 103k
- Forks
- 29.5k
- PR merge metrics
- PR metrics pending
Description
When creating nested tensors on the fly inside a compiled graph using `torch.nested.as_nested_tensor()`, torch.compile fails with an AssertionError in the symbolic shape engine. This breaks workflows that need to run flash attention with mixed dense and nested tensors.
## Versions
PyTorch version: 2.9.0+cu128
## To Reproduce
```python
import torch
import torch.nn as nn
import torch.nn.functional as F
# We need a version of PyTorch with NestedTensor support
# This example assumes PyTorch 2.2 or newer.
if not hasattr(torch, "nested"):
raise ImportError("This example requires a PyTorch version with NestedTensor support.")
def get_sample_nested_tensor() -> torch.Tensor:
"""Creates a sample NestedTensor for testing."""
# A list of tensors with different sequence lengths.
t1 = torch.randn(4, 16) # Batch element 1: seq_len=4, dim=16
t2 = torch.randn(6, 16) # Batch element 2: seq_len=6, dim=16
t3 = torch.randn(3, 16) # Batch element 3: seq_len=3, dim=16
return torch.nested.nested_tensor([t1, t2, t3], layout=torch.jagged)
class ProblematicModel(nn.Module):
"""
This model is designed to replicate the sequence of operations
we identified as causing the error.
"""
def __init__(self, d_model: int, n_heads: int):
super().__init__()
# A learnable tensor, shaped (1, S_i, D). In a real model, this
# could be a set of learned "inducing points" or attention biases.
self.i = nn.Parameter(torch.randn(1, 5, d_model))
self.n_heads = n_heads
self.proj = nn.Linear(d_model, d_model * n_heads)
self.out_proj = nn.Linear(d_model * n_heads, d_model)
self.norm = nn.LayerNorm(d_model)
def forward(self, x: torch.Tensor):
# The key piece: x.shape[0] is a symbolic value representing the batch size.
batch_size = x.shape[0]
# 1. Expand `self.i` based on the symbolic batch size of `x`.
# `i` is now a regular tensor with a symbolic first dimension.
expanded_i = self.i.expand(batch_size, -1, -1)
# 2. THE LIKELY CULPRIT: Convert the expanded tensor back into a NestedTensor.
# This is a highly complex symbolic operation that likely creates
# an "untraceable" symbolic value, tainting `expanded_i`.
if x.is_nested:
# We need to convert i to nested tensors to allow for flash attention
nested_i = torch.nested.as_nested_tensor(expanded_i, layout=torch.jagged)
else:
nested_i = expanded_i
q, k, v = self.proj(nested_i), self.proj(x), self.proj(x)
q = q.view(*q.shape[:-1], self.n_heads, -1)
k = k.view(*k.shape[:-1], self.n_heads, -1)
v = v.view(*v.shape[:-1], self.n_heads, -1)
q = q.transpose(1, 2).contiguous()
k = k.transpose(1, 2).contiguous()
v = v.transpose(1, 2).contiguous()
x = torch.nn.functional.scaled_dot_product_attention(q, k, v)
x = x.transpose(1, 2).contiguous()
x = x.view(*x.shape[:-2], -1)
x = self.out_proj(x)
# torch._dynamo.graph_break() # Setting a graph break here would avoid the error.
x = self.norm(x)
return x
print("Creating model and nested tensor input...")
model = ProblematicModel(d_model=16, n_heads=4)
model = model.cuda()
# Freeze dropout and other layers for deterministic compilation
model.eval()
sample_input = get_sample_nested_tensor()
sample_input = sample_input.cuda()
print("\nAttempting to compile the model...")
# This is where we expect the error to occur.
try:
compiled_model = torch.compile(model)
print("Compilation succeeded unexpectedly.")
print("Running the compiled model...")
with torch.no_grad():
with torch.autocast("cuda"):
output = compiled_model(sample_input)
print(f"Success! Output type: {type(output)}, Shape: {output.shape}")
except AssertionError as e:
print("\n--- CAUGHT THE EXPECTED ERROR ---")
print("The error occurred inside torch.compile's symbolic shape engine.")
print("This confirms our hypothesis about the problematic sequence of operations.")
print("\nError Details:")
# We can expect to see a similar AssertionError with s25 or similar symbolic variable.
print(e)
except Exception as e:
print(f"An unexpected error occurred: {type(e).__name__}: {e}")
```
## Expected behavior
The model should compile successfully and run with nested tensor inputs, or provide a clear error message about what's not supported.
## Workaround
Adding a `torch._dynamo.graph_break()` before the LayerNorm operation avoids the error, but this is not ideal as it breaks the compilation graph.
## Additional context
The error appears to occur when `torch.nested.as_nested_tensor()` is called on a tensor with symbolic dimensions inside a compiled graph. This creates an "untraceable" symbolic value that causes issues in subsequent operations. Ideally, we would be able to run flash attention with mixed dense and nested tensors without requiring a graph break.
cc @cpuhrsch @jbschlosser @bhosmer @drisspg @soulitzer @davidberard98 @YuqingJ @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @kadeng @amjames @Lucaskabela @jataylo @chenyang78
Contributor guide
Assessment
This issue has not been assessed yet.