[Feat] Usage metering, spending limits and per-tenant cost attribution

Part of #777. **After 3.0.** Builds on the trusted tenant/owner and execution identity established by #778, #774 and #745. Those identifiers are the 3.0 foundation; actual usage collection, accounting and limits belong to this issue.

## In plain terms

CAO agents can incur model and compute costs, and **two requests are nowhere near equal** — one agent might use a few thousand tokens and another a few million. Token-priced APIs, provider credits, subscriptions, local models and the mock provider do not all have the same billing model.

On your laptop that's your own bill and your own problem. Across an organisation it becomes somebody's budget, and questions immediately follow: which team is spending this? Can one person's runaway agent burn the quarter's budget in a night? Nobody can answer today.

Microsoft's guidance names this exact trap: *"you often make an implicit assumption that requests are approximately equal… Often that's not the case."* For agents it is emphatically not the case.

## What already exists

CAO has optional orchestration telemetry and predefined token-usage attribute names, but **no wired cross-provider token-usage measurement or attributable spend accounting**.

`telemetry/semconv.py:22-25` defines the `gen_ai.usage.*` names; definitions are not emitted measurements. The existing runtime instrumentation creates an orchestration-dispatch span and increments `cao.orchestration.dispatches` (`services/terminal_service.py:2357-2384`, `telemetry/metrics.py:23-42`). Installing or enabling the optional exporter does not add token collection.

Some adapters recognize numeric-looking terminal output for status or response extraction: Kiro credits, OMP's input/output/cache row, and other providers' context percentages or spinners. They do not turn those displays into structured usage records. Context occupancy is not cumulative billable usage, credits are not automatically currency or tokens, and repeatedly redrawing the same footer is not additional consumption.

Native provider refusals and external extensions are separate capabilities. Grok's weekly-limit refusal is recognized as an error, and operator MCP mentions an external `cao_quota` plugin; neither establishes built-in tenant/person spend accounting. Do not assume that an uninspected provider or plugin has no metering, or that CAO already collects it.

## What this delivers

**Collect usage with a defined source and meaning.** Record provider/source version, units, measured-versus-estimated status, and the execution to which an observation belongs. Establish whether values are incremental or cumulative, how cache usage relates to input totals, and how duplicate observations and resets are handled. Missing or unsupported coverage remains unavailable, not zero.

**Attribute and aggregate it within the tenant boundary.** Use #774/#778/#745's trusted tenant, owner and execution context, not agent-supplied names. Preserve the acting identity separately where delegation or administrative action matters, and define the charging policy rather than assuming the actor and resource owner are always the same person.

**Keep accounting evidence deliberately.** Define retention, collection failures, late observations and the period for which coverage is available. Optional telemetry may export accounting results, but is not by itself a complete, durable ledger. The 3.0 identity foundation does not imply that earlier runs already have recoverable token or billing records.

**Spending limits that actually bite.** OWASP's API Security Top 10 (API4:2023) is direct about this: *"Configure spending limits for all service providers/API integrations. When setting spending limits is not possible, billing alerts should be configured instead."* An alert is the fallback, not the goal.

Define concurrent admission, in-flight work, measurement delay and any possible overshoot. Connect stopping behavior to #745's requested-versus-confirmed cancellation contract. A delayed estimate or billing alert must not be presented as a hard monetary cap; document the enforceable mechanism and unavailable-coverage behavior for each supported provider.

**Compute limits too.** Agents consume CPU, memory and execution slots. Basic workload resource bounds remain in #745; this issue adds tenant-level quota policy. Kubernetes `ResourceQuota` applies within a namespace, while `LimitRange` supplies object-level bounds/defaults. The current broker creates workers in one configured namespace, so adding these objects alone does not create separate CAO-tenant budgets. Define how tenant identity maps to enforceable quotas, for example through namespace allocation or tenant-aware admission/accounting; do not force a namespace-per-tenant redesign into 3.0 by implication.

Network limits need their own supported mechanism. The documented Kubernetes bandwidth plugin is experimental; that is a limit of that mechanism, not proof that no per-workload bandwidth control exists.

**Practical precision, clearly labelled.** Azure identifies overengineering precision as an antipattern and allows estimation where appropriate. Document the pricing/allocation basis, currency and effective period separately from usage. Subscription allocations, rounded displays and infrastructure estimates must not be labelled as exact provider charges.

## Honesty about the sources

Authoritative guidance **does** address token consumption per tenant. Microsoft's [Multitenancy and Azure OpenAI](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/service/openai) explicitly recommends tracking each tenant's token usage in the application when a model deployment is shared, and discusses tenant quotas and cost allocation. The earlier claim that no such guidance exists was wrong.

That is useful design guidance, not a ready-made CAO implementation: coverage and enforcement must be established across the supported provider CLIs. It does not require CAO to adopt Azure OpenAI.

## Acceptance criteria

- [ ] Supported usage sources have documented units, provider/version provenance and measured/estimated/unavailable states. Unavailable usage is never reported as zero.
- [ ] Aggregation is tenant- and principal-scoped, with execution attribution and an explicit charging policy for delegated or administrator-initiated work.
- [ ] Incremental/cumulative values, cache inclusion, duplicate observations, resets and late/missing reports have exercised semantics; terminal redraws are not blindly summed.
- [ ] An administrator can see their own tenant's usage, and only their own.
- [ ] Collection start/coverage, retention and recording failures are explicit; optional telemetry alone is not claimed as a complete historical ledger.
- [ ] A configurable spending limit has defined concurrent-admission, in-flight-work and breach behavior, including provider coverage and possible overshoot. Billing alerts are the documented fallback where enforcement is unavailable.
- [ ] Compute quotas are mapped to actual CAO tenants and exercised with competing tenants, including the shared-namespace case where supported. Per-pod limits alone are not called aggregate tenant quotas.
- [ ] Pricing/allocation and precision trade-offs are documented, and estimates remain distinguishable from measured usage and provider charges.
- [ ] On a laptop none of this is switched on.

## Evidence

| Claim | Where |
| --- | --- |
| Token attribute names exist; current CAO instrumentation counts dispatches instead | `telemetry/semconv.py:22-25`; `services/terminal_service.py:2357-2384`; `telemetry/metrics.py:23-42`; `docs/otel-deployment.md` |
| Metric-looking provider text is used for lifecycle/response boundaries, not structured accounting | `providers/kiro_cli.py:91-92,638-647`; `providers/omp.py:38-41,207-268,297-309`; `providers/codex.py:67-77`; no structured CAO usage producer was found across the adapters |
| Native limits/extensions are not a built-in CAO tenant ledger | `providers/grok_cli.py:739-772`; `ops_mcp_server/server.py:858-864` mentions external `cao_quota`, not a bundled implementation |
| The existing memory injection cap is characters, not model tokens or money | `api/main.py:7258` |
| Worker resources currently share the broker's configured namespace | `examples/cao-clusters/kubernetes/eks/broker.py:58,362-365,1294-1298` |
| Namespace quotas differ from per-object limits and application tenants | [Kubernetes multi-tenancy guidance](https://kubernetes.io/docs/concepts/security/multi-tenancy/) |
| Spending limits guidance | OWASP API Security Top 10, API4:2023 |
| Guidance explicitly covers per-tenant model-token usage | [Microsoft: Multitenancy and Azure OpenAI](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/service/openai), shared-instance and quota/cost-management sections |
| Precision antipattern; unequal requests | Azure Architecture Center, multitenant cost allocation |
| The documented Kubernetes bandwidth plugin is experimental | Kubernetes multi-tenancy documentation; other mechanisms need their own support assessment |

Repository observations are based on `main` at `29b235cf62ed0f9d624bc9ad9afce09ab72f8ddf`; native metering outside CAO's adapters and unbundled plugin implementations were not assessed.

Start with telemetry/semconv.py, telemetry/metrics.py, and services/terminal_service.py to understand the existing instrumentation, then inspect the cited provider adapters and examples/cao-clusters/kubernetes/eks/broker.py. Map the acceptance criteria to supported usage sources, tenant attribution, durable accounting, spending limits, and compute quotas. Done means the documented semantics are implemented and exercised without enabling metering on laptops.

Written by the indexing model from the issue text.