[Release/2.14] E2E performance drops compared with pt2.13

Open

#4,904 opened on Aug 11, 2026 8 comments 0 reactions 6 assignees Claimed by @kaileiyx View on GitHub

hw: PVC performance

Dominant language: Python
Stars: 113
Forks: 128
Avg merge: 5d 9h
Merged PRs (30d): 112

Description

### 🐛 Describe the bug

# Performance Regression

Detailed Drops

## Issue Summary
Performance regression analysis identified multiple models with significant performance degradation. Models with **Ind Ratio < 0.9** or **Eager Ratio < 0.9** are experiencing more than 10% performance drop compared to baseline targets. Analysis reveals that **Eager mode** is severely affected across most models (70+ models), while **Inductor mode** shows relatively better performance (only ~8 models with issues).

## Problematic Model(s)

### Critical Issues (Ratio < 0.75)

| Suite | Dtype | Mode | Model | Ind Ratio | Eager Ratio | Issue Type |
|-------|-------|------|-------|-----------|-------------|------------|
| huggingface | float16 | inference | Qwen/Qwen3-0.6B | 0.7064 | 0.7060 | **CRITICAL**: Both Modes |
| huggingface | amp_bf16 | inference | GPT2ForSequenceClassification | 0.9998 | 0.6511 | **CRITICAL**: Eager |
| torchbench | amp_bf16 | inference | hf_Albert | 1.0087 | 0.6952 | **CRITICAL**: Eager |
| timm_models | amp_bf16 | inference | convnextv2_nano.fcmae_ft_in22k_in1k | 1.0134 | 0.7121 | **CRITICAL**: Eager |
| torchbench | amp_bf16 | inference | timm_vision_transformer | 1.0106 | 0.7333 | **CRITICAL**: Eager |
| huggingface | amp_fp16 | inference | GPT2ForSequenceClassification | 1.0260 | 0.7060 | **CRITICAL**: Eager |
| huggingface | amp_fp16 | inference | MobileBertForQuestionAnswering | 1.0876 | 0.7039 | **CRITICAL**: Eager |
| huggingface | amp_fp16 | inference | RobertaForQuestionAnswering | 0.9724 | 0.7446 | **CRITICAL**: Eager |
| timm_models | amp_fp16 | inference | convnextv2_nano.fcmae_ft_in22k_in1k | 1.0115 | 0.7091 | **CRITICAL**: Eager |
| torchbench | amp_fp16 | inference | hf_Albert | 0.9996 | 0.6694 | **CRITICAL**: Eager |
| torchbench | amp_fp16 | inference | timm_vision_transformer | 1.0133 | 0.7410 | **CRITICAL**: Eager |

### Severe Issues (Ind Ratio < 0.9)

| Suite | Dtype | Mode | Model | Ind Ratio | Eager Ratio | Issue Type |
|-------|-------|------|-------|-----------|-------------|------------|
| huggingface | amp_fp16 | training | MobileBertForQuestionAnswering | 0.8859 | 0.8382 | **SEVERE**: Mixed |
| torchbench | bfloat16 | training | vgg16 | 0.8877 | 1.0008 | **SEVERE**: Inductor |
| torchbench | amp_fp16 | training | vgg16 | 0.8860 | 1.0054 | **SEVERE**: Inductor |
| torchbench | amp_bf16 | inference | hf_Bert_large | 0.9126 | 0.8847 | Mild Mixed |
| huggingface | amp_bf16 | inference | MBartForConditionalGeneration | 0.9725 | 0.8549 | Mild Mixed |
| huggingface | amp_bf16 | inference | RobertaForQuestionAnswering | 0.9673 | 0.7421 | **SEVERE**: Mixed |
| huggingface | amp_bf16 | inference | PLBartForConditionalGeneration | 0.9812 | 0.8343 | Eager |
| huggingface | amp_bf16 | training | MBartForConditionalGeneration | 1.0490 | 0.8491 | Eager |

### Moderate Eager Regression (0.75-0.85) - Selected Examples

| Suite | Dtype | Mode | Model | Ind Ratio | Eager Ratio |
|-------|-------|------|-------|-----------|-------------|
| huggingface | amp_bf16 | inference | AlbertForMaskedLM | 1.0010 | 0.7657 |
| huggingface | amp_bf16 | inference | AlbertForQuestionAnswering | 0.9990 | 0.7620 |
| huggingface | amp_bf16 | inference | BertForQuestionAnswering | 1.0095 | 0.7738 |
| huggingface | amp_bf16 | inference | DistilBertForQuestionAnswering | 1.0073 | 0.7679 |
| huggingface | amp_bf16 | inference | ElectraForQuestionAnswering | 0.9916 | 0.7483 |
| timm_models | amp_bf16 | inference | crossvit_9_240 | 1.0064 | 0.7745 |
| timm_models | amp_bf16 | inference | deit_base_distilled_patch16_224 | 1.0025 | 0.7617 |
| torchbench | amp_bf16 | inference | hf_GPT2 | 0.9618 | 0.7289 |
| torchbench | amp_fp16 | inference | hf_DistilBert | 0.9420 | 0.8076 |
| torchbench | amp_fp16 | inference | hf_GPT2_large | 0.9947 | 0.7709 |

*Note: Over 60+ models show moderate Eager regression (ratio 0.75-0.85)*

## Analysis

### Affected Ratios Summary
- [x] **Ind Ratio < 0.9** - Inductor performance regression (8 models affected)
- [x] **Eager Ratio < 0.9** - Eager mode performance regression (70+ models affected)

## Environment Information

- **PyTorch Version:** 0163eec5ea4
- **OS:** Ubuntu 24.04

## Expected Behavior

- Both Inductor and Eager modes should maintain performance within 10% (ratio ≥ 0.9) of baseline targets
- Eager mode should not show 15-30% performance degradation across 60%+ of models
- Models should perform consistently across different dtypes and test suites

## Actual Behavior

- **Eager mode**: 73 out of 115 models (64%) show performance regression of >10%
- **Inductor mode**: Only 8 out of 115 models (7%) show performance regression
- **Severity**: Many affected models show 15-30% performance drops in Eager mode
- **Consistency**: Issues observed across multiple dtypes, suites, and model types

## Reproduction Steps

To reproduce these performance issues:

```bash
pip install --pre torch==2.14.0.dev20260804+xpu torchaudio==2.11.0.dev20260805+xpu torchvision==0.29.0.dev20260805+xpu triton-xpu==3.7.2+git5fcc14d9 --index-url https://download.pytorch.org/whl/nightly/xpu
TORCH_COMMIT_ID=$(python -c 'import torch; print(torch.version.git_version)')
git clone -b https://github.com/pytorch/pytorch pytorch
cd pytorch
pip install requests pandas psutil scipy pyyaml accelerate
# huggingface
pip install -r .ci/docker/ci_commit_pins/huggingface-requirements.txt || pip install transformers==4.54.0 soxr==0.5.0
# timm_models
TIMM_COMMIT_ID="$(cat .ci/docker/ci_commit_pins/timm.txt 2> /dev/null || echo 'v1.0.14')"
pip install git+https://github.com/huggingface/pytorch-image-models@${TIMM_COMMIT_ID}
# torchbench
TORCHBENCH_COMMIT_ID="$(cat .ci/docker/ci_commit_pins/torchbench.txt 2> /dev/null || echo 'e03a63be')"
cd ../
rm -rf ./benchmark
git clone https://github.com/pytorch/benchmark benchmark
cd benchmark
git checkout ${TORCHBENCH_COMMIT_ID}
python install.py --continue_on_fail

# Example benchmark commands for critical models
python benchmarks/dynamo/huggingface.py --performance -d xpu -n10 --backend=inductor --cold-start-latency --inference --amp --amp-dtype bfloat16 --only

python benchmarks/dynamo/timm_models.py --performance -d xpu -n10 --backend=inductor --cold-start-latency --inference --amp --amp-dtype bfloat16 --only

python benchmarks/dynamo/torchbench.py --performance -d xpu -n10 --backend=inductor --cold-start-latency --inference --amp --amp-dtype bfloat16 --only

```

## Additional Context

### Related Models with Similar Issues

**Models by Suite and Severity:**

**huggingface suite:**
- Severe (ratio < 0.75): Qwen, GPT2ForSequenceClassification, MobileBertForQuestionAnswering, RobertaForQuestionAnswering
- Moderate (ratio 0.75-0.85): AlbertForMaskedLM, BertForQuestionAnswering, DistilBertForQuestionAnswering

**timm_models suite:**
- Severe (ratio < 0.75): convnextv2_nano.fcmae_ft_in22k_in1k
- Moderate (ratio 0.75-0.85): crossvit_9_240, deit_base_distilled_patch16_224, visformer_small

**torchbench suite:**
- Severe (ratio < 0.75): hf_Albert, timm_vision_transformer, vgg16 (Inductor)
- Moderate (ratio 0.75-0.85): hf_GPT2, hf_T5, hf_T5_base

### Logs / Output

Click to expand performance data summary

```
Summary:
- Total models: 115
- Inductor Fails: 8 models (7%)
- Eager Fails: 73 models (64%)
- Critical Issues (<0.75): 11 models (10%)
- Both modes affected: 1 model (Qwen - 0.7064)

Most Common Issue Patterns:
1. Eager mode regression while Inductor performs normally (70+ models)
2. Models with 20-30% performance drops in Eager mode
3. Consistent issues across multiple test suites and dtypes
```

### Versions

torch 2.14.0.dev20260804+xpu
torchao 0.19.0.dev20260810+xpu
torchaudio 2.11.0.dev20260805+xpu
torchvision 0.29.0.dev20260805+xpu

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

[Release/2.14] E2E performance drops compared with pt2.13

Description

Contributor guide

Assessment

Get new issues in your inbox