huggingface / huggingface/optimum-intel

OpenVINO export can silently produce an IR that cannot use PagedAttention (no fused SDPA op)

Open
#1,948 0 comments 1 reaction 1 assignee Claimed by @rkazants View on GitHub
Dominant language
Jupyter Notebook
Stars
620
Forks
270
Avg merge
3d 10h
Merged PRs (30d)
22

Description

### What happens

An OpenVINO export can come out with **decomposed** attention (matmul + softmax) rather than a fused `ScaledDotProductAttention` node. Such an IR loads and generates correctly, but `SDPAToPagedAttention` has nothing to rewrite, so openvino_genai's continuous-batching backend cannot be built:

```
No ScaledDotProductAttention operation observed in the graph,
cannot perform the SDPAToPagedAttention transformation.
```

The practical consequence is the **silent loss of prefix caching** — which for agent workloads is the difference between prefilling a fixed system prompt once and prefilling it every turn.

Nothing at export time warns about this, and nothing in the resulting artifact's name, size, precision or config hints at it. It is only discoverable by grepping the IR or reading a runtime log line.

### Concrete instance

Intel's three published Gemma 4 exports, all `Gemma4ForConditionalGeneration`:

| IR | SDPA ops | SoftMax ops | CB backend |
|---|---|---|---|
| `OpenVINO/gemma-4-E2B-it-int4-ov` | 35 (= layers) | 0 | builds |
| `OpenVINO/gemma-4-26b-a4b-it-int4-ov` | 30 (= layers) | 0 | builds |
| `OpenVINO/gemma-4-E4B-it-int8-ov` | **0** | **42** | **refuses** |

```bash
grep -c 'ScaledDotProductAttention' openvino_language_model.xml
```

Re-exporting the same `google/gemma-4-E4B-it` weights at the same INT8 precision with a current stack (optimum-intel `2.2.0.dev0+e1de5be`, transformers `5.5.4`, OpenVINO 2026.3) produces 42 fused SDPA ops and a working CB backend, byte-identical answers on a 7-case probe set, same 7.8 GB. So the model and the exporter are both capable — that build simply came out decomposed.

Measured cost on an Arc Pro B60 with a ~7.9k-token repeated prefix: 3.03 s per turn without caching vs 1.16 s with, i.e. ~2.6x steady-state TTFT.

### Why it can recur

`FORCE_ATTN_MODEL_CLASSES` in `optimum/exporters/openvino/__main__.py` pins the attention implementation for exactly three model types:

```python
FORCE_ATTN_MODEL_CLASSES = {"phi3_v": "eager", "gemma2": "sdpa", "llama4": "sdpa"}
```

`gemma2` and `llama4` are pinned *to* `sdpa` precisely because, per the adjacent comment, "some models force flash_attn attention by default". Every architecture **not** on that list takes whatever the export environment resolves to — so this is a whack-a-mole list, and any new family is exposed until someone notices and adds it. `gemma4` is currently not listed.

### Suggestions

1. **Add `gemma4`** to `FORCE_ATTN_MODEL_CLASSES` (fixes this family).
2. **More valuable: warn at export time.** After tracing a decoder, if the graph contains no `ScaledDotProductAttention` op, emit a warning that the artifact will not support PagedAttention / prefix caching. That is a cheap static check and it catches every future architecture rather than the ones somebody remembered. Turning it into an opt-out error for `text-generation`/`image-text-to-text` tasks would be stronger still.

Happy to open a PR for either if the direction is agreeable.

Reported alongside a re-export request on the affected model repo: https://huggingface.co/OpenVINO/gemma-4-E4B-it-int8-ov/discussions/1

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.