intel / intel/torch-xpu-ops

[OOB] use sdpa fusion in eager models

Open
#3,778 0 comments 0 reactions 1 assignee Claimed by @jianyizh View on GitHub
test: oob
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

change and upstream model script to enable sdpa in the following models in inference. Torch compile has fused them by pattern matcher.

| # | Model
|---|-------|
| 1 | volo_d1_224_bs2048 |
| 2 | tnt_s_patch16_224_bs2048 |
| 3 | LayoutLMForSequenceClassification_bs256 |
| 4 | hf_T5_large_bs32 |
| 5 | hf_T5_bs16 |
| 6 | T5ForConditionalGeneration_bs32 |
| 7 | T5Small_bs32 |
| 8 | hf_T5_generate_bs16 |
| 9 | visformer_small_bs2048 |
| 10 | hf_T5_base_bs1 |
| 11 | MT5ForConditionalGeneration_bs64 |

**Summary**: 11 model(s) could use `F.scaled_dot_product_attention` in eager code.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.