microsoft / microsoft/winml-cli

swin / image-classification: all models pass wmk perf

Open
#129 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dev experience model / task scale NPU P2 QDQ triaged
Dominant language
Python
Stars
40
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
50

Description

Summary

microsoft/swin-large-patch4-window7-224 fails during QNN compilation with Failed to compose Qnn graph. The ONNX export and quantization complete successfully, but the QNN backend cannot compose the quantized graph.

Eval Results (2026-03-11)

Status Model Task Error
FAIL microsoft/swin-large-patch4-window7-224 image-classification QNN Failed to compose Qnn graph during compilation
PASS microsoft/beit-base-patch16-224-pt22k-ft22k image-classification
PASS google/vit-base-patch16-224 image-classification

1/1 swin model fails.

Error from stderr:

qnn_model.cc:139 onnxruntime::qnn::QnnModel::ComposeGraph Failed to compose Qnn graph.
Suggestion: Check error details above

The ONNX export produces Run outputs: [array([[[[0., 0., 0., ...]]])] before the compilation step — indicating the export is valid but the QNN graph composition fails on the quantized ONNX.

Root Cause

Swin Transformer uses shifted window attention with dynamic window partitioning, which generates ONNX operators that may not be fully supported by the QNN execution provider. The swin-large variant may have graph patterns or operator types that QNN cannot compose, while smaller variants might work.

Note: microsoft/swin-large is a large variant (192M params) — the issue may be size-related or operation-specific (window roll ops, complex reshape patterns).

Current State

  • No swin.py in modelkit/models/hf/
  • No optimization config registered for swin model type
  • QNN compilation failure — not an ONNX export failure

Desired State

microsoft/swin-large-patch4-window7-224 (and other swin variants) pass wmk perf.

Acceptance Criteria

  • microsoft/swin-large-patch4-window7-224 passes wmk perf
  • Fix is universal — no hardcoded swin-specific logic (CLAUDE.md Cardinal Rule #1)
  • uv run pytest tests/ passes (CLAUDE.md Cardinal Rule #3)

Technical Notes

  • First, diagnose which QNN ops fail by enabling verbose QNN logging: check if it's a specific operator (e.g., GridSample, RoiAlign, window-roll ops) or graph size issue
  • If specific ops are unsupported by QNN, consider: (1) disabling those ops via graph optimization before compilation, or (2) falling back to CPU for unsupported subgraphs
  • Swin uses torch.roll for shifted windows — this may export as a complex scatter/gather pattern in ONNX that QNN doesn't support
  • Check if swin-base or swin-tiny pass — if they do, the issue is large-model specific

Related Files

  • modelkit/compiler/ — QNN compilation pipeline
  • modelkit/optim/ — graph optimization configs
  • modelkit/models/hf/__init__.py — where new swin.py import would be added
  • eval_results/2026-03-11/models/microsoft__swin-large-patch4-window7-224__image-classification/result.json

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with wmk perf and enabling verbose QNN logging. Inspect modelkit/compiler/ and modelkit/optim/ to identify the unsupported operator or graph-size limitation, comparing swin variants where possible. Done means the listed Swin model passes, the fix is universal, and uv run pytest tests/ passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.