microsoft / microsoft/winml-cli

mpnet / fill-mask + feature-extraction + sentence-similarity: all models pass wmk perf

Open
#127 1 comment 0 reactions 1 assignee View on GitHub

@DingmaomaoBJTU is already working on this.

Since Mar 31, 2026.

dev experience model / task scale NPU P1 triaged
Dominant language
Python
Stars
40
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
50

Description

Summary

MPNet models fail on fill-mask and sentence-similarity with index out of range in self during ONNX export or inference. Feature-extraction also fails on one model due to a Windows symlink issue in the HF cache.

Eval Results (2026-03-11)

Status Model Task Error
FAIL sentence-transformers/all-mpnet-base-v2 fill-mask index out of range in self
FAIL sentence-transformers/all-mpnet-base-v2 sentence-similarity index out of range in self
FAIL sentence-transformers/multi-qa-mpnet-base-dot-v1 fill-mask index out of range in self
FAIL sentence-transformers/all-mpnet-base-v2 feature-extraction Model not found: model.safetensors (exit 3 — Windows symlink)
FAIL sentence-transformers/multi-qa-mpnet-base-dot-v1 feature-extraction Error parsing message with type 'onnx.ModelProto'
FAIL sentence-transformers/paraphrase-multilingual-mpnet-base-v2 feature-extraction Error parsing message with type 'onnx.ModelProto'
FAIL sentence-transformers/multi-qa-mpnet-base-dot-v1 sentence-similarity Error parsing message with type 'onnx.ModelProto'
FAIL sentence-transformers/paraphrase-multilingual-mpnet-base-v2 sentence-similarity Error parsing message with type 'onnx.ModelProto'
FAIL intfloat/multilingual-e5-large sentence-similarity Error parsing message with type 'onnx.ModelProto'

9/9 models fail — 0 pass.

Root Cause

Two distinct issues:

  1. index out of range in self: MPNet's ONNX export produces a graph with a tensor indexing operation that goes out of bounds. Likely a sequence length or vocab size mismatch in the dummy input generator — MPNet uses a different positional embedding scheme than BERT.
  2. onnx.ModelProto size: Multilingual MPNet models are large (>2GB ONNX) — same issue as xlm-roberta (#429). Fix: use external data format for large exports.
  3. Windows symlink (exit 3): HF hub cache symlinks broken on Windows without Developer Mode — environment issue, not a ModelKit code bug.

Current State

  • No mpnet.py in modelkit/models/hf/ — MPNet relies on Optimum defaults
  • modelkit/models/hf/roberta.py exists (RoBERTa config) but MPNet has a different architecture
  • No ONNX config registered for mpnet model type

Desired State

All 9 MPNet models pass wmk perf (excluding the Windows-symlink-only failure which is environment).

Acceptance Criteria

  • sentence-transformers/all-mpnet-base-v2 passes fill-mask and sentence-similarity
  • sentence-transformers/multi-qa-mpnet-base-dot-v1 passes fill-mask, feature-extraction, and sentence-similarity
  • sentence-transformers/paraphrase-multilingual-mpnet-base-v2 passes feature-extraction and sentence-similarity
  • Fix is universal — no hardcoded model names (CLAUDE.md Cardinal Rule #1)
  • uv run pytest tests/ passes (CLAUDE.md Cardinal Rule #3)

Technical Notes

  • For index out of range: debug MPNet's dummy input shape — MPNet uses max_position_embeddings=514 (514 not 512) and a custom positional embedding offset. The dummy input generator may generate sequence lengths that cause an off-by-one in the embedding lookup.
  • For ONNX size: same fix as xlm-roberta — enable use_external_data_format=True for models above size threshold
  • Consider adding modelkit/models/hf/mpnet.py with a MpnetIOConfig similar to BertIOConfig

Related Files

  • modelkit/models/hf/bert.py — pattern for custom ONNX config with max_position_embeddings
  • modelkit/export/io.pyregister_onnx_overwrite()
  • modelkit/models/hf/__init__.py — where new mpnet.py import would be added
  • eval_results/2026-03-11/models/sentence-transformers__all-mpnet-base-v2__fill-mask/result.json

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.