microsoft / microsoft/winml-cli

Add a config-level reproduction recipe for the Qwen3 genai bundle under examples/recipes/

Open
#1,091 1 comment 0 reactions 1 assignee View on GitHub

@xieofxie is already working on this.

Since Jul 27, 2026.

enhancement P2 triaged
Dominant language
Python
Stars
40
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
50

Description

Summary

Add a recipe under examples/recipes/ that lets users reproduce the Qwen3 (Qwen/Qwen3-0.6B) onnxruntime-genai bundle build at the config level, mirroring the existing per-(model, task, precision) recipe convention.

Deferred from the genai-bundle PR (#1081) to keep that PR focused; capturing the design notes here so the follow-up can execute directly.

Background

winml build -m Qwen/Qwen3-0.6B --ep qnn --device npu|auto -o out/bundle emits a genai bundle. It is composed of 4 ONNX components plus a code-driven assembly step:

Component model_type task device precision
ctx.onnx (transformer prefill) qwen3_transformer_only (decoder_prefill sub-model, seq=64) text-generation NPU/QNN w8a16
iter.onnx (transformer decode) qwen3_transformer_only (decoder_gen sub-model, seq=1) text-generation NPU/QNN w8a16
embeddings.onnx qwen3_embeddings_only feature-extraction CPU fp32
lm_head.onnx qwen3_lm_head_only feature-extraction CPU w4a32
genai_config.json + tokenizer assembly (recipe assemble step)

The 4 components each have a WinMLBuildConfig (config-expressible). The final assembly (genai_config.json, QNN stage session options, ONNX passes) is driven by the registered GenaiBundleRecipe in src/winml/modelkit/models/hf/qwen3/genai.py — it is not a WinMLBuildConfig, so it cannot be captured as a recipe JSON.

Proposed deliverable

Create examples/recipes/Qwen_Qwen3-0.6B/ containing:

  • Per-component WinMLBuildConfig JSONs following the existing role-suffix convention (cf. microsoft_trocr-*/image-to-text_fp16_config_{decoder,encoder}.json):
    • text-generation_w8a16_config_context.json
    • text-generation_w8a16_config_iterator.json
    • feature-extraction_fp32_config_embeddings.json
    • feature-extraction_w4a32_config_lm_head.json
  • A README.md that maps components → bundle, gives the one-command bundle build (assembly), and shows how to rebuild/inspect a single component with winml build -c <config>.json.
  • Add the Qwen3 rows to the top-level examples/recipes/README.md model table.

Also cross-link docs/samples/qwen3-genai-bundle.md.

Design notes / gotchas for the implementer

  • Faithful generation: generate the component configs from code (e.g. winml config --model-type <component> --task <t> --device <d> --ep <ep> --precision <p>) rather than hand-authoring — the transformer configs are large (~28 layers of past/present KV tensors).
  • Transformer max_cache_len override: the composite defaults max_cache_len to max_position_embeddings (40960 for Qwen3-0.6B), but the bundle recipe pins max_cache_len=2048 / prefill_seq_len=64 (see QWEN3_GENAI_BUNDLE_RECIPE in genai.py). To faithfully reproduce the default bundle, the two transformer configs must reflect max_cache_len=2048 (and prefill seq=64 / decode seq=1). Decide whether to bake 2048 in or document the override — winml config uses a single --shape-config for both composite sub-models, so per-sub-model seq differentiation comes from the model classes' _default_seq_len (64/1) automatically.
  • lm_head precision detail: the w4a32 lm_head is finalized to block_size=32, accuracy_level=4 by Qwen3LMHeadOnlyQuantFinalizer (int8 MLAS GEMM). The regression where this fell back to 128/0 was fixed in #1081 (commit preserving model_type through the weight-only precision policy). The recipe README is a good place to note this so users can verify their rebuilt lm_head.onnx matches.
  • Assembly is not config-expressible: make the README explicit that the per-component configs reproduce the 4 ONNX graphs, while the bundle directory (genai_config.json + layout) is produced by the one-command winml build (or scripts/qwen3.py export).

Acceptance criteria

  • examples/recipes/Qwen_Qwen3-0.6B/ with the 4 component configs + README.
  • Configs load via winml build -c and are faithful to the default bundle (transformer max_cache_len=2048; lm_head w4a32/block_size=32).
  • Qwen3 rows added to examples/recipes/README.md.
  • README documents component→bundle mapping, the one-command assembly, and per-component rebuild.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.