Implement --compare for winml perf: side-by-side benchmark across all device+EP combinations

Summary

Add --compare to winml perf to run a benchmark across all available device+EP combinations in a single invocation and display a side-by-side comparison table with a recommendation. Currently users must run winml perf separately for each device/EP and mentally compare results.

Context

A GPU may expose multiple EPs (e.g. OpenVINO and TensorRT RTX). A simple device-level flag would miss these variants — the comparison must be at the device+EP level. --compare captures this correctly.

Note: winml perf already has a --compare-devices stub at perf.py:1126 that prints "not yet implemented". --compare supersedes this stub.

Current Pain

# User must run three (or more) separate commands and mentally compare:
winml perf -m ProsusAI/finbert --device npu
winml perf -m ProsusAI/finbert --device gpu --ep openvino
winml perf -m ProsusAI/finbert --device gpu --ep nv_tensorrt_rtx
winml perf -m ProsusAI/finbert --device cpu

Proposed UX

winml perf -m ProsusAI/finbert --compare          # auto-detect all available device+EP combos
winml perf -m ProsusAI/finbert --compare npu,gpu  # limit to specific devices

Side-by-side output table

  Model   ProsusAI/finbert · text-classification · 100 iterations
  ┌─────────┬─────────────────┬───────────┬──────────┬──────────┬────────────┬─────────┬─────────────────┐
  │ Device  │ EP              │ Mean (ms) │ P50 (ms) │ P99 (ms) │ Throughput │ Memory  │                 │
  ├─────────┼─────────────────┼───────────┼──────────┼──────────┼────────────┼─────────┼─────────────────┤
  │ NPU     │ QNN             │    18.8   │   18.1   │   24.3   │  53 sps    │ 142 MB  │ ✓ Recommended   │
  │ GPU     │ OpenVINO        │    27.4   │   26.9   │   61.2   │  36 sps    │ 389 MB  │                 │
  │ GPU     │ TensorRT RTX    │    22.1   │   21.8   │   38.5   │  45 sps    │ 512 MB  │                 │
  │ CPU     │ CPU EP          │    94.3   │   93.1   │  102.8   │  11 sps    │ 210 MB  │                 │
  └─────────┴─────────────────┴───────────┴──────────┴──────────┴────────────┴─────────┴─────────────────┘
  ✓ NPU · QNN  recommended: lowest latency (18.8 ms), lowest tail variance (p99 24.3 ms)

Recommendation logic

• Primary: lowest mean latency
• Tiebreaker: lowest p99 tail (penalises GPU jitter from thermal throttling and compositor interrupts)

Unavailable combinations are omitted with a note:

  NPU  skipped — no QNN EP detected on this system

Design Notes

• Auto-detection. --compare with no value queries the EP registry for all available providers and maps them to device+EP rows.
• Explicit filter. --compare npu,gpu limits to rows matching the listed devices.
• EP variants per device. A single device may appear in multiple rows if multiple EPs are available for it (e.g. GPU row for OpenVINO + GPU row for TensorRT RTX).
• Sequential runs. Benchmarks run sequentially to avoid thermal/memory contention skewing results.
• --compare supersedes --compare-devices stub. Remove or alias the old stub.
• --compare is mutually exclusive with --device / --ep (those flags select a single target).
• JSON report. When --report is used alongside --compare, results are written as a per-combination array (see #443).

Acceptance Criteria

• --compare (no value) auto-detects and benchmarks all available device+EP combinations
• --compare npu,gpu limits comparison to rows matching those devices
• Multiple EP variants for the same device each appear as separate rows
• Table shows: device, EP, mean latency, p50, p99, throughput, peak memory, recommended marker
• Recommended marker (✓ Recommended) is inline in the same row as the best combination
• Summary line below the table names the recommended combination with key metrics
• Unavailable combinations are skipped with an explanatory note, not a hard error
• --compare is mutually exclusive with --device / --ep
• Old --compare-devices stub is removed or aliased to --compare
• Results written to --report JSON as a per-combination array when --compare is active
• Unit tests cover auto-detection, table formatting, EP-variant grouping, and recommendation logic

Related

• #155 — wmk perf expand metrics (p50/p95/p99 prerequisite for the comparison table)
• #443 — --report JSON flag (comparison results must be captured in the report)
• #445 — --monitor GPU utilization (per-combination monitor should fire during each run)