microsoft / microsoft/winml-cli
SA: Coverage report investigation — QNN NPU (2026-04-01)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 40
- Forks
- 11
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 50
Description
Overview
Investigate regressions and unknown ops surfaced in the QNN NPU SA coverage report (2026-04-01 run). Report: https://icy-moss-029643d00.6.azurestaticapps.net/sa_e2e_coverage/QNNExecutionProvider_NPU/0401/sa_eval_report.html
Summary stats: Avg SUPPORTED (Pre): 95.0% → (Post): 94.9% | Avg delta: -0.1% | Avg UNKNOWN: 27 | All-SUPPORTED (Post): 98.3%
Regressions (Post < Pre)
| Model | Task | Pre | Post | Delta |
|---|---|---|---|---|
| microsoft/swin-large-patch4-window7-224 | image-classification | 68.8% | 66.7% | -2.1% |
| PekingU/rtdetr_v2_r18vd | object-detection | 76.7% | 74.2% | -2.5% |
| facebook/bart-large-mnli | text-classification | 77.3% | 76.2% | -1.1% |
Partial op patterns involved: Reshape, Transpose, Pad, Slice, Add, Mul, Sub, TopK
Unknown Ops
| Model | Unknown Count | Op Types |
|---|---|---|
| facebook/bart-large-mnli | 1 | GatherND |
| PekingU/rtdetr_r50vd_coco_o365 | 2 | GridSample, Tile |
| PekingU/rtdetr_r101vd_coco_o365 | 2 | GridSample, Tile |
| PekingU/rtdetr_v2_r18vd | 2 | GridSample, Tile |
Acceptance Criteria
- Root cause identified for swin-large regression (-2.1%)
- Root cause identified for rtdetr_v2_r18vd regression (-2.5%)
- Root cause identified for bart-large-mnli regression (-1.1%)
-
GatherND,GridSample,Tileadded to QNN coverage DB or flagged as unsupported - Next report shows 0 regressions and 0 unknown ops for the above models
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked QNN NPU SA coverage report for the 2026-04-01 run and compare the listed regressions with their partial op patterns. Investigate the three named models and the unknown GatherND, GridSample, and Tile operations. Done means root causes are identified, the operations are covered or explicitly unsupported, and a subsequent report has no listed regressions or unknown ops.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100