pytorch / pytorch/pytorch

[Regression] 20x slower vision model forward pass in PyTorch 2.9.1 vs 2.8.0

Open
#172,537 5 comments 1 reaction 0 assignees View on GitHub
high priority module: performance triaged
Dominant language
Python
Stars
103k
Forks
29.5k
PR merge metrics
PR metrics pending

Description

### 🐛 Describe the bug

Vision model forward pass is ~20x slower in PyTorch 2.9.1 compared to 2.8.0 when processing video/image inputs. Text-only inference is unaffected.

**Performance comparison (same GPU, same model, same inputs):**

| Operation | PyTorch 2.8.0 | PyTorch 2.9.1 | Regression |
|-----------|---------------|---------------|------------|
| Text-only forward | 0.08s | 0.08s | None |
| Vision forward pass | 0.68s | 14.15s | **~20x slower** |
| Generate 50 tokens (vision) | 1.44s | 15.06s | **~10x slower** |

The slowdown is isolated to the forward pass when vision inputs (pixel_values) are provided. Text-only generation works identically on both versions.

### Minimal Reproducible Example

```python
import torch
import time
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor

MODEL_NAME = "Qwen/Qwen3-VL-8B" # Or any Qwen3-VL model

model = Qwen3VLForConditionalGeneration.from_pretrained(
MODEL_NAME,
device_map="cuda:0",
torch_dtype=torch.bfloat16,
attn_implementation="sdpa", # Same issue with flash_attention_2
)
model.eval()
processor = AutoProcessor.from_pretrained(MODEL_NAME)

# Test with video/image input
conversation = [
{
"role": "user",
"content": [
{"type": "video", "video": "path/to/any/video.mp4"},
{"type": "text", "text": "Describe this video."},
],
}
]

inputs = processor.apply_chat_template(
conversation,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt"
).to("cuda:0")

# Warmup
with torch.no_grad():
_ = model(**inputs, return_dict=True)
torch.cuda.synchronize()

# Timed forward pass
with torch.no_grad():
torch.cuda.synchronize()
t0 = time.time()
outputs = model(**inputs, return_dict=True)
torch.cuda.synchronize()
print(f"Forward pass time: {time.time()-t0:.2f}s")

# Expected: ~0.7s (PyTorch 2.8.0)
# Actual: ~14s (PyTorch 2.9.1)
```

### Versions

Slow environment (PyTorch 2.9.1):

PyTorch: 2.9.1+cu126
Triton: 3.5.1
CUDA: 12.6
transformers: 4.57.3
GPU: [YOUR GPU MODEL]

Fast environment (PyTorch 2.8.0):

PyTorch: 2.8.0+cu126
Triton: 3.4.0
CUDA: 12.6
transformers: 4.57.3
GPU: [SAME GPU MODEL]

cc @ezyang @gchanan @kadeng @msaroufim @jerryzh168

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.