microsoft / microsoft/winml-cli

dinov2 / image-feature-extraction: all models pass wmk perf

Open
#138 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

model / task scale P2 triaged
Dominant language
Python
Stars
40
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
50

Description

Summary

facebook/dinov2-giant fails with Error parsing message with type 'onnx.ModelProto' — the giant DINOv2 model exceeds the protobuf 2GB limit. Smaller variants (small, base, large) all pass.

Eval Results (2026-03-11)

Status Model Task Error
PASS facebook/dinov2-small image-feature-extraction
PASS facebook/dinov2-base image-feature-extraction
FAIL facebook/dinov2-large image-feature-extraction TIMEOUT (600s)
FAIL facebook/dinov2-giant image-feature-extraction Error parsing message with type 'onnx.ModelProto'
PASS facebook/dino-vitb16 image-feature-extraction
PASS facebook/dino-vits16 image-feature-extraction
PASS StanfordAIMI/dinov2-base-xray-224 image-feature-extraction
PASS microsoft/rad-dino image-feature-extraction

2/8 fail — only large/giant variants.

Root Cause

dinov2-giant (~1.1B params) generates an ONNX file exceeding 2GB. Same root cause as xlm-roberta (#429) — requires save_as_external_data=True during export.

dinov2-large (~307M params) times out — may be a compilation timeout on the larger attention graph.

Current State

  • No dinov2.py in modelkit/models/hf/ — relies on Optimum defaults
  • Small/base/xray/rad-dino all pass → export path is functional for smaller models
  • External data format not enabled for large models

Desired State

All DINOv2 variants pass wmk perf, including dinov2-giant.

Acceptance Criteria

  • facebook/dinov2-giant passes wmk perf
  • facebook/dinov2-large passes wmk perf
  • Existing small/base/dino-vitb16/dino-vits16 variants continue to pass
  • Fix is size-based (universal) — not dinov2-specific (CLAUDE.md Cardinal Rule #1)
  • uv run pytest tests/ passes (CLAUDE.md Cardinal Rule #3)

Technical Notes

  • Fix: enable use_external_data_format=True for exports above ONNX size threshold — coordinate with #429 (xlm-roberta) as the same fix applies
  • For dinov2-large TIMEOUT: may need an increased timeout for larger ViT models, or optimization to speed up QNN compilation for large attention graphs
  • This issue will be automatically resolved once the shared external-data-format fix from #429 is implemented

Related Files

  • modelkit/export/config.pyWinMLExportConfig, check for use_external_data_format field
  • eval_results/2026-03-11/models/facebook__dinov2-giant__image-feature-extraction/result.json

References

  • Blocked by / coordinate with: #429 (xlm-roberta ONNX size fix — same root cause and fix)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with modelkit/export/config.py and inspect WinMLExportConfig for external-data support, then review issue #429 and the referenced result.json. Run the DINOv2 wmk perf evaluations and uv run pytest tests/. Done means large and giant pass while the existing smaller variants remain passing and the fix is size-based.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.