microsoft / microsoft/winml-cli

[Task] sentence-similarity EP coverage investigation

Open
#297 0 comments 0 reactions 1 assignee View on GitHub

@ssss141414 is already working on this.

Since Apr 9, 2026.

model / task scale P2 triaged
Dominant language
Python
Stars
40
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
50

Description

Overview

Sentence similarity models produce dense sentence embeddings optimized for semantic textual similarity tasks (STS). These include Sentence-Transformers and BAAI/BGE variants. All 10 models pass on at least one EP.

This issue focuses on the 3 models with incomplete EP coverage — all exhibiting the same VitisAI failure pattern. The affected models are the multilingual MiniLM and two smaller BGE variants, suggesting the failure may be related to model size or multilingual tokenizer handling on VitisAI.

Agent Scenarios

  • Semantic deduplication agent: identify and merge duplicate or near-duplicate records (support tickets, product listings, knowledge base articles) by comparing sentence embeddings
  • Cross-lingual search agent: retrieve semantically similar content across languages using multilingual models (paraphrase-multilingual-*) for global enterprise search
  • RAG similarity scoring agent: compute relevance between retrieved passages and a query to filter or rerank before LLM generation

EP Coverage Status

Model QNN OV VitisAI
sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 PASS PASS FAIL
BAAI/bge-small-en-v1.5 PASS PASS FAIL
BAAI/bge-base-en-v1.5 PASS PASS FAIL

Reference: all-MiniLM-L6-v2, all-mpnet-base-v2, bge-m3, paraphrase-multilingual-mpnet-base-v2, bge-large-en-v1.5, multi-qa-mpnet-base-dot-v1, multilingual-e5-large already pass all 3 EPs.

Acceptance Criteria

  • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 passes all 3 EPs (investigate VitisAI failure)
  • BAAI/bge-small-en-v1.5 passes all 3 EPs (investigate VitisAI failure)
  • BAAI/bge-base-en-v1.5 passes all 3 EPs (investigate VitisAI failure)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.