microsoft / microsoft/winml-cli

Add zero-shot object detection schema and evaluator to winml eval

Open
#1,147 1 comment 0 reactions 1 assignee View on GitHub

@DingmaomaoBJTU is already working on this.

Since Jul 27, 2026.

enhancement P2 triaged
Dominant language
Python
Stars
40
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
50

Description

Problem

winml eval has no schema or evaluator for zero-shot-object-detection, so valid open-vocabulary detectors such as OWLv2 cannot produce a task metric. Reusing the existing fixed-label object-detection evaluator is not semantically valid: zero-shot detection must derive text queries from category names, post-process grounded detections, and map query-local results back to dataset category IDs.

Reproduction

$MODEL_ID = 'google/owlv2-base-patch16-finetuned'
$REV = 'd69b086b07123308a4e198343ab2824e1af7774a'
$SNAPSHOT = 'checkpoint_snapshot'
python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='$MODEL_ID', revision='$REV', local_dir='$SNAPSHOT')"
winml eval -m model.onnx --model-id $SNAPSHOT --task zero-shot-object-detection --dataset detection-datasets/coco --dataset-revision cf0b22332314a937e9dc8a1957b21725430bb41d --split val --samples 1 --no-shuffle --streaming --ep cpu --device cpu --output eval.json --overwrite --format json --no-color

Actual result:

Error: Evaluation failed: Task 'zero-shot-object-detection' is not supported.

The task is absent from both the evaluation task schema and evaluator registry, so execution stops before dataset or model evaluation.

Expected capability

Add a generalized zero-shot-object-detection evaluation schema/evaluator that:

  1. derives candidate text queries from dataset category names;
  2. runs the model processor's grounded object-detection post-processing;
  3. maps query-local detections to dataset category IDs; and
  4. reports a standard detection metric such as COCO mAP.

This should remain distinct from fixed-label object-detection evaluation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.