microsoft / microsoft/winml-cli
Add inpainting evaluator, dataset schema, and canonical metric
@DingmaomaoBJTU is already working on this.
Since Jul 27, 2026.
- Dominant language
- Python
- Stars
- 40
- Forks
- 11
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 50
Description
Problem
winml eval does not register the canonical inpainting task, although runtime inference can use an image + binary mask contract.
Both eval --schema --task inpainting and artifact evaluation reject the task before dataset loading. This prevents task-metric evidence for direct-ONNX inpainting models such as opencv/inpainting_lama.
Requested capability
- Register an
inpaintingevaluator and schema. - Define the required image, mask, and ground-truth restored-target columns.
- Document a canonical metric or metric set for inpainting quality.
- Keep input preprocessing aligned with the runtime image + mask contract.
Reproduction
winml eval --schema --task inpainting
Observed with WinML CLI 0.2.0:
Task 'inpainting' is not supported by winml eval.
A public image+mask dataset can exercise input shape, but a valid evaluator also needs authoritative restored targets and a documented metric; scores must not be inferred from an input-only dataset.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.