[Regression] 20x slower vision model forward pass in PyTorch 2.9.1 vs 2.8.0
- Dominant language
- Python
- Stars
- 103k
- Forks
- 29.5k
- PR merge metrics
- PR metrics pending
Description
### 🐛 Describe the bug
Vision model forward pass is ~20x slower in PyTorch 2.9.1 compared to 2.8.0 when processing video/image inputs. Text-only inference is unaffected.
**Performance comparison (same GPU, same model, same inputs):**
| Operation | PyTorch 2.8.0 | PyTorch 2.9.1 | Regression |
|-----------|---------------|---------------|------------|
| Text-only forward | 0.08s | 0.08s | None |
| Vision forward pass | 0.68s | 14.15s | **~20x slower** |
| Generate 50 tokens (vision) | 1.44s | 15.06s | **~10x slower** |
The slowdown is isolated to the forward pass when vision inputs (pixel_values) are provided. Text-only generation works identically on both versions.
### Minimal Reproducible Example
```python
import torch
import time
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
MODEL_NAME = "Qwen/Qwen3-VL-8B" # Or any Qwen3-VL model
model = Qwen3VLForConditionalGeneration.from_pretrained(
MODEL_NAME,
device_map="cuda:0",
torch_dtype=torch.bfloat16,
attn_implementation="sdpa", # Same issue with flash_attention_2
)
model.eval()
processor = AutoProcessor.from_pretrained(MODEL_NAME)
# Test with video/image input
conversation = [
{
"role": "user",
"content": [
{"type": "video", "video": "path/to/any/video.mp4"},
{"type": "text", "text": "Describe this video."},
],
}
]
inputs = processor.apply_chat_template(
conversation,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt"
).to("cuda:0")
# Warmup
with torch.no_grad():
_ = model(**inputs, return_dict=True)
torch.cuda.synchronize()
# Timed forward pass
with torch.no_grad():
torch.cuda.synchronize()
t0 = time.time()
outputs = model(**inputs, return_dict=True)
torch.cuda.synchronize()
print(f"Forward pass time: {time.time()-t0:.2f}s")
# Expected: ~0.7s (PyTorch 2.8.0)
# Actual: ~14s (PyTorch 2.9.1)
```
### Versions
Slow environment (PyTorch 2.9.1):
PyTorch: 2.9.1+cu126
Triton: 3.5.1
CUDA: 12.6
transformers: 4.57.3
GPU: [YOUR GPU MODEL]
Fast environment (PyTorch 2.8.0):
PyTorch: 2.8.0+cu126
Triton: 3.4.0
CUDA: 12.6
transformers: 4.57.3
GPU: [SAME GPU MODEL]
cc @ezyang @gchanan @kadeng @msaroufim @jerryzh168
Contributor guide
Assessment
This issue has not been assessed yet.