huggingface / huggingface/optimum-intel
OpenVINO export can silently produce an IR that cannot use PagedAttention (no fused SDPA op)
- Dominant language
- Jupyter Notebook
- Stars
- 620
- Forks
- 270
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 22
Description
### What happens
An OpenVINO export can come out with **decomposed** attention (matmul + softmax) rather than a fused `ScaledDotProductAttention` node. Such an IR loads and generates correctly, but `SDPAToPagedAttention` has nothing to rewrite, so openvino_genai's continuous-batching backend cannot be built:
```
No ScaledDotProductAttention operation observed in the graph,
cannot perform the SDPAToPagedAttention transformation.
```
The practical consequence is the **silent loss of prefix caching** — which for agent workloads is the difference between prefilling a fixed system prompt once and prefilling it every turn.
Nothing at export time warns about this, and nothing in the resulting artifact's name, size, precision or config hints at it. It is only discoverable by grepping the IR or reading a runtime log line.
### Concrete instance
Intel's three published Gemma 4 exports, all `Gemma4ForConditionalGeneration`:
| IR | SDPA ops | SoftMax ops | CB backend |
|---|---|---|---|
| `OpenVINO/gemma-4-E2B-it-int4-ov` | 35 (= layers) | 0 | builds |
| `OpenVINO/gemma-4-26b-a4b-it-int4-ov` | 30 (= layers) | 0 | builds |
| `OpenVINO/gemma-4-E4B-it-int8-ov` | **0** | **42** | **refuses** |
```bash
grep -c 'ScaledDotProductAttention' openvino_language_model.xml
```
Re-exporting the same `google/gemma-4-E4B-it` weights at the same INT8 precision with a current stack (optimum-intel `2.2.0.dev0+e1de5be`, transformers `5.5.4`, OpenVINO 2026.3) produces 42 fused SDPA ops and a working CB backend, byte-identical answers on a 7-case probe set, same 7.8 GB. So the model and the exporter are both capable — that build simply came out decomposed.
Measured cost on an Arc Pro B60 with a ~7.9k-token repeated prefix: 3.03 s per turn without caching vs 1.16 s with, i.e. ~2.6x steady-state TTFT.
### Why it can recur
`FORCE_ATTN_MODEL_CLASSES` in `optimum/exporters/openvino/__main__.py` pins the attention implementation for exactly three model types:
```python
FORCE_ATTN_MODEL_CLASSES = {"phi3_v": "eager", "gemma2": "sdpa", "llama4": "sdpa"}
```
`gemma2` and `llama4` are pinned *to* `sdpa` precisely because, per the adjacent comment, "some models force flash_attn attention by default". Every architecture **not** on that list takes whatever the export environment resolves to — so this is a whack-a-mole list, and any new family is exposed until someone notices and adds it. `gemma4` is currently not listed.
### Suggestions
1. **Add `gemma4`** to `FORCE_ATTN_MODEL_CLASSES` (fixes this family).
2. **More valuable: warn at export time.** After tracing a decoder, if the graph contains no `ScaledDotProductAttention` op, emit a warning that the artifact will not support PagedAttention / prefix caching. That is a cheap static check and it catches every future architecture rather than the ones somebody remembered. Turning it into an opt-out error for `text-generation`/`image-text-to-text` tasks would be stronger still.
Happy to open a PR for either if the direction is agreeable.
Reported alongside a re-export request on the affected model repo: https://huggingface.co/OpenVINO/gemma-4-E4B-it-int8-ov/discussions/1
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.