[OOB] use sdpa fusion in eager models
Open
test: oob
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 112
Description
change and upstream model script to enable sdpa in the following models in inference. Torch compile has fused them by pattern matcher.
| # | Model
|---|-------|
| 1 | volo_d1_224_bs2048 |
| 2 | tnt_s_patch16_224_bs2048 |
| 3 | LayoutLMForSequenceClassification_bs256 |
| 4 | hf_T5_large_bs32 |
| 5 | hf_T5_bs16 |
| 6 | T5ForConditionalGeneration_bs32 |
| 7 | T5Small_bs32 |
| 8 | hf_T5_generate_bs16 |
| 9 | visformer_small_bs2048 |
| 10 | hf_T5_base_bs1 |
| 11 | MT5ForConditionalGeneration_bs64 |
**Summary**: 11 model(s) could use `F.scaled_dot_product_attention` in eager code.
Contributor guide
Assessment
This issue has not been assessed yet.