intel / intel/torch-xpu-ops

[Release/2.13][PVC]E2E performance drops vs pytorch 2.12 release

Open
#4,032 10 comments 0 reactions 2 assignees Assigned to @jianyizh View on GitHub
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 13h
Merged PRs (30d)
107

Description

### 🐛 Describe the bug

PT2.13 validation on PVC E2E performance drops compared with pytorch 2.12 release after applying rerun verification.
## Affected Cases

Category | Name | XPU vs. Baseline [Eager] | XPU vs. Baseline [Inductor]
-- | -- | -- | --
torchbench_bfloat16_inference | basic_gnn_sage | 1.033048225 | 0.889459247
timm_models_amp_bf16_training | convmixer_768_32 | 1.065230856 | 0.896586761
timm_models_float16_inference | eca_botnext26ts_256 | 0.972208062 | 0.894399155
timm_models_amp_bf16_training | ese_vovnet19b_dw | 1.006515646 | 0.899445939
timm_models_amp_fp16_training | ese_vovnet19b_dw | 0.998275614 | 0.89123017
timm_models_bfloat16_training | ese_vovnet19b_dw | 0.971472581 | 0.828204808
timm_models_amp_bf16_inference | gluon_inception_v3 | 0.996142511 | 0.893312881
timm_models_amp_fp16_inference | gluon_inception_v3 | 0.997478885 | 0.895848038
timm_models_amp_bf16_inference | hrnet_w18 | 0.958880163 | 0.895765696
timm_models_amp_fp16_inference | inception_v3 | 1.002681724 | 0.89741462
timm_models_bfloat16_inference | inception_v3 | 0.892464045 | 0.850637078
timm_models_float16_inference | inception_v3 | 0.895569982 | 0.83934736
torchbench_amp_bf16_inference | llama_v2_7b_16h | 1.005475714 | 0.861781275
timm_models_amp_bf16_inference | mixnet_l | 0.915816301 | 0.865424147
timm_models_amp_fp16_inference | mixnet_l | 0.911219938 | 0.836249932
torchbench_amp_bf16_training | mobilenet_v2 | 0.996434362 | 0.787474689
torchbench_amp_fp16_training | mobilenet_v2 | 0.997915524 | 0.798097421
torchbench_bfloat16_inference | mobilenet_v2 | 0.99440247 | 0.896927125
torchbench_bfloat16_training | mobilenet_v2 | 0.991810565 | 0.656428325
torchbench_float16_training | mobilenet_v2 | 0.98805625 | 0.643161617
torchbench_float32_training | mobilenet_v2 | 1.005220842 | 0.76998421
timm_models_amp_bf16_training | mobilevit_s | 0.989497537 | 0.86574512
timm_models_amp_fp16_training | mobilevit_s | 0.997000351 | 0.868868128
torchbench_amp_fp16_inference | moondream | 1.009566654 | 0.862009324
torchbench_float32_training | nvidia_deeprecommender | 0.900701174 | 0.892796187
huggingface_float16_inference | OPTForCausalLM | 0.883895656 | 0.881292518
timm_models_bfloat16_training | resmlp_12_224 | 0.998772964 | 0.850621449
torchbench_amp_bf16_inference | resnext50_32x4d | 0.899259071 | 0.86653979
torchbench_amp_fp16_inference | resnext50_32x4d | 0.954679616 | 0.885608079
timm_models_amp_bf16_inference | rexnet_100 | 0.956923843 | 0.897547473
timm_models_amp_fp16_inference | rexnet_100 | 0.972421573 | 0.872069365
timm_models_amp_fp16_inference | selecsls42b | 0.959054003 | 0.895358686
timm_models_bfloat16_inference | spnasnet_100 | 1.004845592 | 0.873191401
torchbench_bfloat16_inference | timm_resnest | 0.990708298 | 0.888805371
timm_models_amp_bf16_inference | tinynet_a | 0.986041986 | 0.897089825
timm_models_amp_fp16_inference | tinynet_a | 0.984326424 | 0.89402526
huggingface_amp_bf16_inference | XLNetLMHeadModel | 1.165826075 | 0.894512559
huggingface_amp_fp16_inference | XLNetLMHeadModel | 1.106054191 | 0.884746875
huggingface_bfloat16_inference | XLNetLMHeadModel | 1.249190471 | 0.859158626
huggingface_float16_inference | XLNetLMHeadModel | 1.210675885 | 0.843354486

### Versions

Pytorch 2.13

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.