intel / intel/torch-xpu-ops

[OOB] models missing sdpa fusion in torch compile inference

Open
#3,779 0 comments 0 reactions 1 assignee Claimed by @jianyizh View on GitHub
test: oob
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 13h
Merged PRs (30d)
107

Description

Some OOB models are missing sdpa fusion in torch.compile, we should investigate because eager mode has this fusion

Models with SDPA ops that are decomposed (BMM+Softmax+BMM) instead of fused (flash attention). Decomposition adds overhead.

**25 model(s)** with decomposed SDPA (out of 49 models with SDPA ops).

| # | Model | SDPA Ops | B580 Status | 4080 Status | G31 Status | Correction (ms) |
|---|-------|--------:|----------|----------|----------|--------:|
| 1 | BertForQuestionAnswering_bs512 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 382.8 |
| 2 | RobertaForQuestionAnswering_bs512 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 382.8 |
| 3 | BartForConditionalGeneration_bs32 | 36 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 362.2 |
| 4 | MBartForConditionalGeneration_bs32 | 36 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 362.2 |
| 5 | AlbertForMaskedLM_bs64 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 255.2 |
| 6 | AlbertForQuestionAnswering_bs64 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 255.2 |
| 7 | ElectraForQuestionAnswering_bs1024 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 255.2 |
| 8 | PLBartForConditionalGeneration_bs32 | 18 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 135.8 |
| 9 | MobileBertForQuestionAnswering_bs4096 | 24 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 134.4 |
| 10 | BertForMaskedLM_bs128 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 95.7 |
| 11 | CamemBert_bs128 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 95.7 |
| 12 | hf_Albert_bs128 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 95.7 |
| 13 | hf_Bart_bs64 | 18 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 72.4 |
| 14 | hf_Bert_large_bs512 | 24 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 71.6 |
| 15 | PegasusForConditionalGeneration_bs256 | 36 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 54.3 |
| 16 | DistilBertForQuestionAnswering_bs2048 | 6 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 53.3 |
| 17 | hf_DistilBert_bs128 | 6 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 47.9 |
| 18 | BlenderbotSmallForConditionalGeneration_bs256 | 24 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 34.3 |
| 19 | M2M100ForConditionalGeneration_bs128 | 36 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 27.8 |
| 20 | RobertaForCausalLM_bs32 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 23.9 |
| 21 | MobileBertForMaskedLM_bs512 | 24 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 17.6 |
| 22 | ElectraForCausalLM_bs64 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 16.0 |
| 23 | PegasusForConditionalGeneration_bs64 | 36 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 14.6 |
| 24 | DistilBertForMaskedLM_bs512 | 6 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 13.5 |
| 25 | DistilBertForQuestionAnswering_bs64 | 6 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 1.9 |

**Summary**: 25 models decomposed on all platforms, 0 XPU-only. Total potential savings: 3262 ms.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.