[OOB] models missing sdpa fusion in torch compile inference
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 107
Description
Some OOB models are missing sdpa fusion in torch.compile, we should investigate because eager mode has this fusion
Models with SDPA ops that are decomposed (BMM+Softmax+BMM) instead of fused (flash attention). Decomposition adds overhead.
**25 model(s)** with decomposed SDPA (out of 49 models with SDPA ops).
| # | Model | SDPA Ops | B580 Status | 4080 Status | G31 Status | Correction (ms) |
|---|-------|--------:|----------|----------|----------|--------:|
| 1 | BertForQuestionAnswering_bs512 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 382.8 |
| 2 | RobertaForQuestionAnswering_bs512 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 382.8 |
| 3 | BartForConditionalGeneration_bs32 | 36 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 362.2 |
| 4 | MBartForConditionalGeneration_bs32 | 36 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 362.2 |
| 5 | AlbertForMaskedLM_bs64 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 255.2 |
| 6 | AlbertForQuestionAnswering_bs64 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 255.2 |
| 7 | ElectraForQuestionAnswering_bs1024 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 255.2 |
| 8 | PLBartForConditionalGeneration_bs32 | 18 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 135.8 |
| 9 | MobileBertForQuestionAnswering_bs4096 | 24 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 134.4 |
| 10 | BertForMaskedLM_bs128 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 95.7 |
| 11 | CamemBert_bs128 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 95.7 |
| 12 | hf_Albert_bs128 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 95.7 |
| 13 | hf_Bart_bs64 | 18 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 72.4 |
| 14 | hf_Bert_large_bs512 | 24 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 71.6 |
| 15 | PegasusForConditionalGeneration_bs256 | 36 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 54.3 |
| 16 | DistilBertForQuestionAnswering_bs2048 | 6 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 53.3 |
| 17 | hf_DistilBert_bs128 | 6 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 47.9 |
| 18 | BlenderbotSmallForConditionalGeneration_bs256 | 24 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 34.3 |
| 19 | M2M100ForConditionalGeneration_bs128 | 36 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 27.8 |
| 20 | RobertaForCausalLM_bs32 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 23.9 |
| 21 | MobileBertForMaskedLM_bs512 | 24 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 17.6 |
| 22 | ElectraForCausalLM_bs64 | 12 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 16.0 |
| 23 | PegasusForConditionalGeneration_bs64 | 36 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 14.6 |
| 24 | DistilBertForMaskedLM_bs512 | 6 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 13.5 |
| 25 | DistilBertForQuestionAnswering_bs64 | 6 | DECOMPOSED | DECOMPOSED | DECOMPOSED | 1.9 |
**Summary**: 25 models decomposed on all platforms, 0 XPU-only. Total potential savings: 3262 ms.
Contributor guide
Assessment
This issue has not been assessed yet.