microsoft / microsoft/winml-cli
Clarify evaluation loaders and unify composite model loading
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 40
- Forks
- 11
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 50
Description
Context
_select_model_loader() in src/winml/modelkit/eval/evaluate.py selects several distinct loading contracts, but _ModelLoaderKind does not document what each kind means or who owns model/session construction.
Mask generation currently selects EVALUATOR_MANAGED: WinMLMaskGenerationEvaluator constructs its encoder and decoder ONNX Runtime sessions directly. This bypasses WinMLAutoModel because the composite model type is not registered. That makes the loading design task-specific and difficult to reuse for future evaluators backed by composite models.
Work
- Document every
_ModelLoaderKindvalue, including its input form, loading owner, returned model shape, build-pipeline behavior, and intended evaluation mode. - Investigate and design an architecture-agnostic composite-model loading path through
WinMLCompositeModel/WinMLAutoModel. - Migrate mask-generation evaluation to the shared composite-model path if feasible.
- Ensure future composite evaluators can reuse the same path without adding task- or architecture-specific branching to
_select_model_loader(). - Add pytest coverage for loader selection and composite-model loading behavior.
Acceptance criteria
- Loader-kind semantics and precedence are clear at their declaration.
- Mask-generation session ownership is either moved into a reusable composite model abstraction or the remaining blocker is explicitly documented.
- The design does not hardcode model architecture, graph node, tensor, or layer names in shared loading code.
- Existing ONNX, ONNX-to-HF comparison, ONNX-to-ONNX comparison, and GenAI loading behavior remains unchanged.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/winml/modelkit/eval/evaluate.py with _select_model_loader() and _ModelLoaderKind, then inspect WinMLCompositeModel and WinMLAutoModel. Add pytest coverage for loader selection and composite-model loading, preserving existing loading behavior; done when loader semantics are documented and mask-generation loading uses a reusable path or its blocker is recorded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100