huggingface / huggingface/optimum-intel
[OpenVINO] Eagle3 tooling: greedy-equality test can't detect a degraded drafter; export needs einops, declared only in TESTS_REQUIRE
- Dominant language
- Jupyter Notebook
- Stars
- 620
- Forks
- 270
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 22
Description
### Context
While benchmarking EAGLE-3 speculative decoding with openvino-genai on Intel Arc 140V (Lunar Lake), I exported community EAGLE-3 heads (`Tengyunw/qwen3_30b_moe_eagle3`, `AngelSlim/Qwen3-a3B_eagle3`) with the Eagle3 export support added in #1588. Two related tooling observations, bundled here as one report. Environment: optimum-intel 2.1.0, optimum 2.3.0, transformers 5.5.4, torch 2.13.0, openvino 2026.3.1, openvino-genai 2026.3.1.0, Python 3.14.4, Windows 11.
### 1. The speculative-decoding test's equality assertion cannot detect a functionally degraded drafter
Eagle3 export is gated by an equality check — greedy output WITH the draft must equal greedy output WITHOUT it:
- v2.1.0: `tests/openvino/test_genai.py:574`, `LLMPipelineWithEagle3TestCase::test_compare_outputs` (assertion at `:632`)
- main: renamed to `LLMPipelineWithSpeculativeDecodingTestCase` (`:577`) and parameterized over Eagle3 **and** DFlash (`SPECULATIVE_DECODING_MODELS`, `:584`); assertion at `:649`
That equality holds for any drafter that loads and runs. The test sets `do_sample=False`, so validation never reaches the probabilistic branch: in openvino-genai 2026.3.1.0, `Sampler::validate_candidate` (`src/cpp/src/sampling/sampler.cpp:1362`) takes the rejection-sampling path only under `do_sample && has_real_probabilities` (`:1378`, acceptance at `:1391`), while the greedy path is a plain argmax comparison at `:1393` — and a rejected candidate is discarded so the main model's own token stands. The emitted sequence is the target's greedy sequence by construction, so an untrained or semantically degraded head yields byte-identical output, just with zero accepted tokens.
To be precise about what the suite *does* catch: a structurally invalid export fails loudly rather than silently. openvino-genai's shared Eagle3 graph transforms assert that exactly three hidden layers are provided (`src/cpp/src/speculative_decoding/eagle3_model_transforms.cpp:71`) and throw if the FC weights cannot be located (`:199`) — both reached from the continuous-batching path this pipeline builds (`src/cpp/src/continuous_batching/pipeline.cpp:92`); `test_decoder.py::test_load_and_infer_with_eagle3_model` (v2.1.0 `:867`, main `:872`) separately covers load-and-infer. The gap is the narrower one: nothing in the suite checks whether the exported head is *effective*.
The discriminating signal is acceptance — e.g. asserting `extended_perf_metrics.get_num_accepted_tokens() > 0` in the speculative run. That method is on `SDPerModelsPerfMetrics` and has been present since `releases/2026/0`, which is the `min_openvino_version` the test already declares for Eagle3.
On whether such an assertion would be stable: the test generates only 10 tokens (`max_new_tokens=10, min_new_tokens=10`) from the prompt `"Paris is the capital of"`, and I have not measured acceptance at that length. What I can report is one run of the test's own model pair (`AngelSlim/Qwen3-1.7B_eagle3` + `Qwen/Qwen3-1.7B`), exported with 2.1.0 and run on openvino-genai 2026.3.1.0 at a 1062-token prompt: `get_num_accepted_tokens()` returned 27 over a 64-token greedy generation, and openvino-genai's own summary for that run printed `AVG acceptance rate, %: 15.1714` with `Generated tokens by draft model: 174`. Comfortably non-zero there; a longer generation in the test, or a threshold set from a measured floor, would be the safer form.
### 2. `einops` is required at runtime by the Eagle3 load path, but declared only in TESTS_REQUIRE
The Eagle3 export routes model loading through optimum's own bundled modeling module via an injected `auto_map`, and transformers' dynamic-module loader then scans that module's imports — failing with ``ImportError: This modeling file requires the following packages that were not found in your environment: einops. Run `pip install einops` `` on a plain `pip install "optimum-intel[openvino]"` environment.
`einops` sits only in `TESTS_REQUIRE` (`setup.py:56`, the same line on v2.1.0 and main), while `EXTRAS_REQUIRE["openvino"]` (`:79`) declares `nncf`, `openvino` and `openvino-tokenizers` only. I checked the dependency closure of the environment that hit this: the only installed distributions declaring `einops` are `optimum` (extras `dev`/`tests`) and `optimum-intel` (extra `tests`), so nothing supplies it transitively. None of the community Eagle3 heads involved ships a `.py` file or an `auto_map` of its own (all three are config + weights + README, `"architectures": ["LlamaForCausalLMEagle3"]`), so the dynamic-module scan is triggered solely by optimum-intel's injection pointing at optimum-intel's own `model_patcher.py`. CI environments have it and user environments do not — which is also why item 1's test never encounters the error. Suggestion: either declare it where the `[openvino]` extra can see it, or make the scan skip it. On the second option — the two `einops` uses in `model_patcher.py` are already function-local (`_qwen_rotate_half`, v2.1.0 `:858` / main `:884`; `_internlm2_attention_forward`, v2.1.0 `:1383` / main `:1409`), and neither is on the Eagle3 path, so deferring the import further does not help: transformers' `get_imports` parses the file with `ast` and recurses into every child node, collecting nested imports too — it skips only `ast.Try` bodies and `is_*_available()`-guarded `ast.If` blocks (`transformers/dynamic_module_utils.py`, checked on 5.5.4). Wrapping those two imports in `try/except ImportError` would clear the scan; a lazy import alone would not.
### Repro sketch (item 2)
```
pip install "optimum-intel[openvino]" torch # note: no einops
optimum-cli export openvino -m Tengyunw/qwen3_30b_moe_eagle3 --task text-generation-with-past \
--weight-format fp16 --disable-convert-tokenizer --trust-remote-code out_dir
# -> ImportError: ... einops ...
pip install einops # then re-run: export succeeds
```
Both fixes look small (an acceptance assert in the speculative-decoding test; a dependency declaration or lazy import). I can open a PR for either or both if that is the preferred shape. Noting that #1926 is currently refactoring this same test class — happy to work on top of it rather than across it.
Related context: the exported heads run into a separate MoE-target limitation on the openvino.genai side, reported at https://github.com/openvinotoolkit/openvino.genai/issues/4390.
Not verified on my side: I did not run the CI suite with a deliberately degraded drafter, and I have not measured acceptance at the test's own 10-token length — the figure above is a single run at a 1062-token prompt on my own machine, outside the CI harness. The `einops` ImportError was observed exporting `Tengyunw/qwen3_30b_moe_eagle3`; the mechanism is head-independent (the import lives in optimum's own bundled module, reached through the injected `auto_map` for any eagle3 architecture), but I did not separately reproduce it on the other head. All observations are from optimum-intel 2.1.0 on Windows / Python 3.14.
AI assistance used: yes
How: the investigation and this report were produced with Claude Code (agent-driven session); I set the direction and reviewed and approved this text before posting.
Human validation performed: the exports and failures were executed on my own machine in the session described; every quoted error traces to preserved logs. I did not independently validate the source analysis beyond review.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with tests/openvino/test_genai.py, especially LLMPipelineWithSpeculativeDecodingTestCase::test_compare_outputs, and review setup.py's TESTS_REQUIRE and EXTRAS_REQUIRE["openvino"]. Check the speculative-decoding metrics behavior and the Eagle3 export dependency path before choosing the fix shape. Done means the test detects an ineffective drafter and a plain openvino installation can export Eagle3 models without the reported einops ImportError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, build-system, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100