[Feature] Model latency self-serve tool — profile workload, assess against latency target, guide next step

Problem

Developers often can't tell whether the latency they see on a model deployment is expected for their workload or a platform issue. Today the options are guesswork or a support case, with no evidence in hand.

Proposal

A self-serve latency flow for a selected deployment + time range:

1. Profile — show deployment context (model/version, offer/SKU, region), traffic shape (input/output tokens P50/P95, streaming ratio, cache usage, request rate over time) and latency experience (TTFT / TBT / TTLT P50/P95, error rate, throttling).
2. Assess — compare observed latency with the latency target for the current offer (same model/version, metric, percentile). Within target → done. Otherwise → compare with a reference benchmark for a similar traffic shape (matched on model/version, offer, streaming, input/output token buckets).
3. Guide next step — based on the result:
   • Traffic shape explains the gap → workload guidance (reduce context, stabilize reusable prefix, smooth request rate)
   • Within target but workload needs lower / more predictable latency → offer guidance (better-fit offer where available)
   • Gap remains unexplained → escalation package (time range, traffic profile, percentiles, benchmark comparison, request IDs)

Scope

• In scope (MVP): workload-level explanation and triage, offer guidance, evidence packaging. Request-level token distributions require Log Analytics opt-in; fields stay unavailable if disabled.
• Out of scope: authoritative request-level API or platform root-cause attribution.

Open questions

• Latency target contract: source, metric, percentile, customer-facing wording
• Offer guidance: when to suggest another offer; which offers/models/regions are eligible
• Release scope: customer-facing, internal preview, or both
• Telemetry gap: Azure Monitor separates audio tokens but not image tokens — need another source for the text/image split

Tracking

Internal: AB#5582548