microsoft / microsoft/onnxruntime-genai

ModelBuilder support use_cache=False and hidden_states_layers

Open
#2,217 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

# Request ModelBuilder support for:

**use_cache=False** — build/run without KV-cache inputs/outputs (encoder-style, not causal decoding).
**hidden_states_layers** — optional outputs for intermediate layer hidden states (layer indices configurable), not only the final hidden state.

Why
Needed for Flux.2-klein-4B text encoder: no autoregressive KV cache, and hidden_states_layers=[9, 18, 27].

Ask
Document the API/flags, align ONNX I/O with these modes, and add a small example or test if possible.

I have address a draft pr: #2211 and https://github.com/microsoft/Olive/pull/2511, It is a doodle version. Please make changes as you like.
Could anyone please help realize the functional in the onnxruntime-genai? Many thanks.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the ModelBuilder API and flags described here, then review draft PR #2211 and the referenced Olive pull request. Trace how ONNX inputs and outputs are defined for cache and hidden-state options; done means documenting both modes and adding the requested example or test for the Flux.2-klein-4B configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai, backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.