Surface HTTP 429 throttling as a structured signal (distinct error code + run summary + Retry-After) so wrapping tools can warn / back off

Summary

When the target agent is rate-limited, runevals / wiqd agent eval returns every scenario as a generic failure with no structured way for a calling tool to tell it was throttling (HTTP 429) vs. a genuine agent error, and surfaces no quota/retry information. A wrapping UI therefore can't warn the user, disable/delay re-runs, or back off — a whole run comes back empty (rubric N/A) after burning time and quota.

Context

We wrap the eval CLI in a desktop app for non-developer reviewers running evals against deployed M365 Copilot declarative agents (WorkIQ A2A transport). Invoked as wiqd agent eval -> m365.eval.cli (v1.11.0).

Observed behavior

During throttling every prompt fails and retries with a fixed backoff:

{"level":"warning","operation":"send-prompt","logger":"m365.eval.cli",
 "message":"Prompt \"When is my next holiday?\": attempt 2/4 failed (HTTP 429); retrying in 4s."}

After retries are exhausted, each result item is:

{
  "prompt": "When is my next holiday?",
  "response": "",
  "status": "error",
  "error": {
    "code": "agentRequestFailed",
    "message": "Agent request failed: A2A request failed (HTTP 429 Too Many Requests)."
  }
}

Problems:

• error.code is generic (agentRequestFailed) — identical to any other agent failure. The only 429 signal is the substring HTTP 429 inside a human-readable message, which is fragile to match on.
• No quota / cooldown info. Verified the WorkIQ A2A endpoint returns a bare 429 — no Retry-After, no RateLimit-* / x-ratelimit-* headers. So the fixed 4s backoff cannot adapt, and there is nothing for the CLI to surface even if it wanted to.
• No run-level throttle summary. A run can come back 16/17 items throttled with no aggregate signal; the wrapper only learns by parsing each item message.

Impact

• A wrapping tool can't distinguish "throttled, retry later" from "the agent is broken", so it can't warn the user, disable/delay the Run button, or auto-retry.
• Non-developer users just see empty responses + N/A rubric and assume their agent or evals are broken.
• Quota/time is burned on runs that are doomed from the first 429.

Requests

1. Structured throttle signal — emit a distinguishable error for rate limiting, e.g. error.code: "rate_limited" (and/or error.http_status: 429, error.retryable: true) on throttled items, so callers can detect it programmatically.
2. Run-level summary — in the results metadata (and CLI output): counts of throttled vs failed vs succeeded items, and a suggested cooldown when known.
3. Honor + surface Retry-After / RateLimit-* when present; otherwise fall back to exponential backoff + jitter with configurable max-retries / base-delay CLI flags. (Today WorkIQ A2A sends none — worth flagging upstream as well.)
4. (Nice to have) a distinct exit code / JSON field for "run degraded/aborted due to throttling" vs genuine eval failures, for CI gating.

Workaround in flight

Lowering --concurrency reduces how hard we trip the limit, but it doesn't give the wrapper any visibility to gate re-runs — hence this request.

Environment

• m365.eval.cli 1.11.0 via wiqd agent eval (deployed declarative agent, WorkIQ A2A).
• Verified bare-429 (no rate-limit headers) against https://workiq.svc.cloud.microsoft/a2a/.agents.

Start at the wiqd agent eval entry point and m365.eval.cli behavior described in the issue, then trace how A2A HTTP 429 responses become item errors and run results. Check how retry information and response metadata are currently handled. Done means callers can identify throttling structurally, see a run-level summary, and receive applicable retry or backoff information.