microsoft / microsoft/winml-cli

Clarify evaluation loaders and unify composite model loading

Open
#1,294 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
40
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
50

Description

Context

_select_model_loader() in src/winml/modelkit/eval/evaluate.py selects several distinct loading contracts, but _ModelLoaderKind does not document what each kind means or who owns model/session construction.

Mask generation currently selects EVALUATOR_MANAGED: WinMLMaskGenerationEvaluator constructs its encoder and decoder ONNX Runtime sessions directly. This bypasses WinMLAutoModel because the composite model type is not registered. That makes the loading design task-specific and difficult to reuse for future evaluators backed by composite models.

Work

  • Document every _ModelLoaderKind value, including its input form, loading owner, returned model shape, build-pipeline behavior, and intended evaluation mode.
  • Investigate and design an architecture-agnostic composite-model loading path through WinMLCompositeModel / WinMLAutoModel.
  • Migrate mask-generation evaluation to the shared composite-model path if feasible.
  • Ensure future composite evaluators can reuse the same path without adding task- or architecture-specific branching to _select_model_loader().
  • Add pytest coverage for loader selection and composite-model loading behavior.

Acceptance criteria

  • Loader-kind semantics and precedence are clear at their declaration.
  • Mask-generation session ownership is either moved into a reusable composite model abstraction or the remaining blocker is explicitly documented.
  • The design does not hardcode model architecture, graph node, tensor, or layer names in shared loading code.
  • Existing ONNX, ONNX-to-HF comparison, ONNX-to-ONNX comparison, and GenAI loading behavior remains unchanged.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/winml/modelkit/eval/evaluate.py with _select_model_loader() and _ModelLoaderKind, then inspect WinMLCompositeModel and WinMLAutoModel. Add pytest coverage for loader selection and composite-model loading, preserving existing loading behavior; done when loader semantics are documented and mask-generation loading uses a reusable path or its blocker is recorded.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.