microsoft / microsoft/onnxruntime-genai
ModelBuilder support use_cache=False and hidden_states_layers
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
# Request ModelBuilder support for:
**use_cache=False** — build/run without KV-cache inputs/outputs (encoder-style, not causal decoding).
**hidden_states_layers** — optional outputs for intermediate layer hidden states (layer indices configurable), not only the final hidden state.
Why
Needed for Flux.2-klein-4B text encoder: no autoregressive KV cache, and hidden_states_layers=[9, 18, 27].
Ask
Document the API/flags, align ONNX I/O with these modes, and add a small example or test if possible.
I have address a draft pr: #2211 and https://github.com/microsoft/Olive/pull/2511, It is a doodle version. Please make changes as you like.
Could anyone please help realize the functional in the onnxruntime-genai? Many thanks.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the ModelBuilder API and flags described here, then review draft PR #2211 and the referenced Olive pull request. Trace how ONNX inputs and outputs are defined for cache and hidden-state options; done means documenting both modes and adding the requested example or test for the Flux.2-klein-4B configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ai, backend-api-design
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100