Feature request: Agent Value V2 — judge agents by outcomes, not usage, using data the Kit already has

Problem statement

User Story

As an administrator, I want the Kit to tell me which agents are delivering outcomes and which are not, so that I can keep, repair or retire them on evidence rather than on recent usage.

Problem

The Kit can tell an administrator whether an agent is risky (Compliance Hub) and whether it has an owner (Agent Inventory), but not whether it is doing any good. The only automatic signal for "still worth keeping" is recent usage, and it is wrong in both directions:

• A quarterly close assistant. Finance uses it two weeks a quarter. It is flagged as dead every quarter.
• A busy helpdesk agent. 800 real conversations; 6 % of users confirm their question was answered; 55 % give up half-way. It is flagged as a success.

The second hides behind a counting rule. Copilot Studio marks a conversation Resolved when the user confirms or when the user goes silent and the session times out; added together, the helpdesk agent reads "90 % resolved". #822 multiplies executions by a value per execution — and an execution the user abandoned delivers none of that value.

Justification

Agent Value V1 classifies what value an agent should deliver from its description; its documentation says V2 "may extend this with session-level telemetry for real-time value estimation". This is that V2. It proposes; the owner decides.

Proposed solution

What the Kit already has

All of this exists today. V2 adds derived aggregates and an owner-declared Purpose Record — no new telemetry source.

• Resolved / escalated / abandoned counts, engaged count, CSAT — Insights Hub, Transcript Metrics
• Run and invocation success rates, tool failures, Copilot Credits per agent — Insights Hub, Agent Usage History
• Who changed what, when (opt-in, 30 days) — Agent Change Tracker
• Type, behaviour and value category per agent — Agent Value (V1)
• Publish state, quarantine, knowledge sources, business unit, open cases — Agent Inventory, Compliance Hub

Three additions

1. Split "resolved". Three columns on Transcript Metrics — Resolved Confirmed Count, Resolved Implied Count, Survey Response Count — filled by the flow that already writes the table; Resolved Count stays. Rates are over engaged conversations: confirmedRate = ResolvedConfirmedCount / EngagedCount. Useful on its own, and it lets #822 multiply confirmed outcomes rather than every execution.

2. A Purpose Record per agent, from its business owner: purpose statement, owner, success metric (a choice — Confirmed Resolution Rate, Run Success Rate, Escalation Rate, CSAT, Manual), threshold, expected frequency, review date, and Declared or Inferred. Inferred records are pre-filled from the "simple formula or signal" V1's prompt already generates and discards — a suggestion, not a setting, until the owner confirms. Not on Agent Details, which the daily sync rewrites (Ask 3). The natural-language purpose is not scored directly. It is used to suggest a measurable success metric; verdicts use only owner-confirmed structured fields and existing telemetry.

3. A weekly verdict job — deterministic, aggregates only, no LLM. It first asks whether an agent should be judged at all, then judges it on the owner's declared metric (else the default for its type), against the owner's bar and at least five peers of the same metric, type and behaviour:

Question	Result
Unpublished, quarantined, or under an open compliance case?	Deferred to compliance. Risk before value
Window = max(90 days, 2 × expected frequency) exceeds 365 days?	Not enough evidence — annual agents are never judged, never retired for it
No activity in the window — engaged conversations, runs or invocations, by workload — and no evidence of change (Change Tracker where opted in, Agent Details Modified On / Published On otherwise)?	Retirement candidate — no usage
No Purpose, under 30 conversations (or runs), or no readable metric?	Not enough evidence; owner told once. Purpose still unconfirmed two runs later and under 30 in 180 days → Retirement candidate — unclaimed. A sentence of purpose ends that; a small sample alone never does
Below the owner's bar or well under the peer median? Escalating when that is not its job, hitting token limits often, or costing far more than peers?	Needs repair, with a reason — maker change, dependency decay, unattributed decline, below target, escalation, overflow, cost. Unchanged and unanswered for 60 days → Retirement candidate — unrepaired
Conversations halved while quality held? A published twin in the same business unit sharing a knowledge source?	Needs repair as an owner question — still needed? duplicates? Never counts toward retirement
Otherwise	Keep; top quarter of peers at or below their median cost → Keep and scale (a candidate input to #821's Portfolio View). No bar and no peers → Not enough evidence, not a pass

Metric-based verdicts change only when two consecutive runs agree; compliance signals apply at once. Every retirement candidate becomes a Compliance Hub case of a new type, Value review, pinned to Manual review — Quarantine and Delete are not available for it. V2 never deletes. Before anything acts: two shadow runs, then an agreement gate — cases are enabled only once blind administrator labels agree with the verdicts above 80 %.

Delivery

Increment 1, the split, stands alone. Increment 2, Purpose Record and verdict job, needs 1. Increment 3, a daily aggregate of the Debugger's token signal, is optional.

Three asks

1. Is the confirmed / implied split acceptable on its own, as a first increment?
2. Insights Hub has no components in config/*/customDeploymentSettings.json, unlike ValueSummary|, AgentCompliance| and ConversationKPI|. Which solution owns it, and which flow writes Transcript Metrics? Increment 1 cannot be scoped without them.
3. If the verdict layer is wanted: should the Purpose fields live in #819's attribute framework, on #820's Value Driver table, or in a new Agent Purpose table?

Additional context

Expected behaviour

• A quarterly agent is evaluated over its declared cadence and is not treated as unused merely because the latest quarter is quiet.
• An agent with low confirmed resolution is flagged for repair even when its combined Resolved rate appears high.
• Missing transcripts or an active compliance case never produce a Keep or retirement decision.