microsoft / microsoft/winml-cli
feat: smart precision with device-aware auto defaults
Open
Nobody has claimed this yet.
dev experience
feature scale
GPU
hardware
NPU
P1
QDQ
triaged
- Dominant language
- Python
- Stars
- 40
- Forks
- 11
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 50
Description
Summary
Support `--device auto --precision auto` where auto precision intelligently selects the optimal quantization/conversion strategy based on the target device.
Device-Aware Auto Precision
| Device | Auto Precision | Pipeline Behavior |
|---|---|---|
| NPU | int8 (quantized) | Enable QDQ quantization (uint8/int8) |
| GPU | fp16 (non-LLM) / w4a16 (LLM) | FP16 conversion for standard models; weight-int4 + activation-fp16 for LLMs |
| CPU | fp16 | FP16 conversion, no int quantization |
Explicit Precision Overrides
| `--precision` | Behavior |
|---|---|
| `auto` | Device-dependent (see table above) |
| `fp32` | No conversion — optimize + compile only |
| `fp16` | FP16 conversion, skip int quantization entirely |
| `int8` | QDQ int8 quantization (current default behavior) |
| `int16` | QDQ int16 quantization (#229) |
Design
Pipeline adaptation
- fp16 path: Skip quantize stage entirely, add fp16 conversion step (cast ops or mixed precision)
- int8/int16 path: Run quantize stage as today
- fp32 path: Skip quantize, standard optimize + compile
LLM detection for GPU auto
- Heuristic: decoder-only architecture, large vocab size, model size > threshold
- Or explicit flag: `--model-type llm` to force w4a16 path
- Avoid hardcoding model names — use structural properties
CLI integration
- `wmk build -m model_id -o output/ --device npu --precision auto`
- `wmk build -m model_id -o output/ --device cpu --precision fp16`
- Default: `--device auto --precision auto`
Acceptance Criteria
- `--precision auto` resolves based on `--device` value
- NPU auto → int8 quantization
- GPU auto → fp16 (non-LLM) or w4a16 (LLM)
- CPU auto → fp16
- `--precision fp16` skips quantization stage
- `--precision fp32` skips quantization and conversion
- LLM detection uses structural properties, not hardcoded model names
- Backward compatible — existing behavior unchanged when precision not specified
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.