microsoft / microsoft/winml-cli
mpnet / fill-mask + feature-extraction + sentence-similarity: all models pass wmk perf
Open
@DingmaomaoBJTU is already working on this.
Since Mar 31, 2026.
dev experience
model / task scale
NPU
P1
triaged
- Dominant language
- Python
- Stars
- 40
- Forks
- 11
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 50
Description
Summary
MPNet models fail on fill-mask and sentence-similarity with index out of range in self during ONNX export or inference. Feature-extraction also fails on one model due to a Windows symlink issue in the HF cache.
Eval Results (2026-03-11)
| Status | Model | Task | Error |
|---|---|---|---|
| FAIL | sentence-transformers/all-mpnet-base-v2 | fill-mask | index out of range in self |
| FAIL | sentence-transformers/all-mpnet-base-v2 | sentence-similarity | index out of range in self |
| FAIL | sentence-transformers/multi-qa-mpnet-base-dot-v1 | fill-mask | index out of range in self |
| FAIL | sentence-transformers/all-mpnet-base-v2 | feature-extraction | Model not found: model.safetensors (exit 3 — Windows symlink) |
| FAIL | sentence-transformers/multi-qa-mpnet-base-dot-v1 | feature-extraction | Error parsing message with type 'onnx.ModelProto' |
| FAIL | sentence-transformers/paraphrase-multilingual-mpnet-base-v2 | feature-extraction | Error parsing message with type 'onnx.ModelProto' |
| FAIL | sentence-transformers/multi-qa-mpnet-base-dot-v1 | sentence-similarity | Error parsing message with type 'onnx.ModelProto' |
| FAIL | sentence-transformers/paraphrase-multilingual-mpnet-base-v2 | sentence-similarity | Error parsing message with type 'onnx.ModelProto' |
| FAIL | intfloat/multilingual-e5-large | sentence-similarity | Error parsing message with type 'onnx.ModelProto' |
9/9 models fail — 0 pass.
Root Cause
Two distinct issues:
index out of range in self: MPNet's ONNX export produces a graph with a tensor indexing operation that goes out of bounds. Likely a sequence length or vocab size mismatch in the dummy input generator — MPNet uses a different positional embedding scheme than BERT.onnx.ModelProtosize: Multilingual MPNet models are large (>2GB ONNX) — same issue as xlm-roberta (#429). Fix: use external data format for large exports.- Windows symlink (exit 3): HF hub cache symlinks broken on Windows without Developer Mode — environment issue, not a ModelKit code bug.
Current State
- No
mpnet.pyinmodelkit/models/hf/— MPNet relies on Optimum defaults modelkit/models/hf/roberta.pyexists (RoBERTa config) but MPNet has a different architecture- No ONNX config registered for
mpnetmodel type
Desired State
All 9 MPNet models pass wmk perf (excluding the Windows-symlink-only failure which is environment).
Acceptance Criteria
-
sentence-transformers/all-mpnet-base-v2passes fill-mask and sentence-similarity -
sentence-transformers/multi-qa-mpnet-base-dot-v1passes fill-mask, feature-extraction, and sentence-similarity -
sentence-transformers/paraphrase-multilingual-mpnet-base-v2passes feature-extraction and sentence-similarity - Fix is universal — no hardcoded model names (CLAUDE.md Cardinal Rule #1)
-
uv run pytest tests/passes (CLAUDE.md Cardinal Rule #3)
Technical Notes
- For
index out of range: debug MPNet's dummy input shape — MPNet usesmax_position_embeddings=514(514 not 512) and a custom positional embedding offset. The dummy input generator may generate sequence lengths that cause an off-by-one in the embedding lookup. - For ONNX size: same fix as xlm-roberta — enable
use_external_data_format=Truefor models above size threshold - Consider adding
modelkit/models/hf/mpnet.pywith aMpnetIOConfigsimilar toBertIOConfig
Related Files
modelkit/models/hf/bert.py— pattern for custom ONNX config with max_position_embeddingsmodelkit/export/io.py—register_onnx_overwrite()modelkit/models/hf/__init__.py— where new mpnet.py import would be addedeval_results/2026-03-11/models/sentence-transformers__all-mpnet-base-v2__fill-mask/result.json
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.