microsoft / microsoft/winml-cli

Add inpainting evaluator, dataset schema, and canonical metric

Open
#1,158 1 comment 0 reactions 1 assignee View on GitHub

@DingmaomaoBJTU is already working on this.

Since Jul 27, 2026.

enhancement P2 triaged
Dominant language
Python
Stars
40
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
50

Description

Problem

winml eval does not register the canonical inpainting task, although runtime inference can use an image + binary mask contract.

Both eval --schema --task inpainting and artifact evaluation reject the task before dataset loading. This prevents task-metric evidence for direct-ONNX inpainting models such as opencv/inpainting_lama.

Requested capability

  • Register an inpainting evaluator and schema.
  • Define the required image, mask, and ground-truth restored-target columns.
  • Document a canonical metric or metric set for inpainting quality.
  • Keep input preprocessing aligned with the runtime image + mask contract.

Reproduction

winml eval --schema --task inpainting

Observed with WinML CLI 0.2.0:

Task 'inpainting' is not supported by winml eval.

A public image+mask dataset can exercise input shape, but a valid evaluator also needs authoritative restored targets and a documented metric; scores must not be inferred from an input-only dataset.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.