microsoft / microsoft/winml-cli
Add zero-shot object detection schema and evaluator to winml eval
@DingmaomaoBJTU is already working on this.
Since Jul 27, 2026.
- Dominant language
- Python
- Stars
- 40
- Forks
- 11
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 50
Description
Problem
winml eval has no schema or evaluator for zero-shot-object-detection, so valid open-vocabulary detectors such as OWLv2 cannot produce a task metric. Reusing the existing fixed-label object-detection evaluator is not semantically valid: zero-shot detection must derive text queries from category names, post-process grounded detections, and map query-local results back to dataset category IDs.
Reproduction
$MODEL_ID = 'google/owlv2-base-patch16-finetuned'
$REV = 'd69b086b07123308a4e198343ab2824e1af7774a'
$SNAPSHOT = 'checkpoint_snapshot'
python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='$MODEL_ID', revision='$REV', local_dir='$SNAPSHOT')"
winml eval -m model.onnx --model-id $SNAPSHOT --task zero-shot-object-detection --dataset detection-datasets/coco --dataset-revision cf0b22332314a937e9dc8a1957b21725430bb41d --split val --samples 1 --no-shuffle --streaming --ep cpu --device cpu --output eval.json --overwrite --format json --no-color
Actual result:
Error: Evaluation failed: Task 'zero-shot-object-detection' is not supported.
The task is absent from both the evaluation task schema and evaluator registry, so execution stops before dataset or model evaluation.
Expected capability
Add a generalized zero-shot-object-detection evaluation schema/evaluator that:
- derives candidate text queries from dataset category names;
- runs the model processor's grounded object-detection post-processing;
- maps query-local detections to dataset category IDs; and
- reports a standard detection metric such as COCO mAP.
This should remain distinct from fixed-label object-detection evaluation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.