microsoft / microsoft/winml-cli

feat(eval): support automatic speech recognition metrics

Open
#1,229 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement model / task scale P2 triaged
Dominant language
Python
Stars
40
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
50

Description

Feature gap

winml eval does not register automatic-speech-recognition, so valid Wav2Vec2 CTC artifacts cannot be evaluated with a task metric through the CLI.

Reproduction

  • Model: MahmoudAshraf/mms-300m-1130-forced-aligner
  • Task: automatic-speech-recognition
  • Artifacts: verified CPU FP32 and FP16 ONNX models with input_values [1,16000] -> logits [1,49,31]

Run either:

winml eval --schema --task automatic-speech-recognition

or:

winml eval -m <model.onnx> --model-id MahmoudAshraf/mms-300m-1130-forced-aligner --task automatic-speech-recognition --device cpu --ep cpu

Both are rejected because ASR is absent from the supported-task registry.

Requested support

Add an ASR evaluator/schema with raw-audio preprocessing and a defensible transcript/audio dataset contract. For CTC checkpoints, expose an appropriate metric such as WER; forced-alignment checkpoints may additionally need alignment/timestamp quality evaluation.

The referenced model card only says “forced alignment dataset” and does not identify a dataset repository, subset, split, or revision, so callers must be able to provide an explicit evaluation dataset rather than silently substituting one.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supported-task registry used by winml eval --schema and reproduce the rejection with the commands in the issue. Define the explicit audio/transcript dataset contract, add ASR schema and raw-audio evaluation with an appropriate CTC metric such as WER, and verify that both schema discovery and model evaluation accept the task without silently selecting a dataset.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.