[Bug]: LLM cost budget is enforced as a continuously-refilling rate (GCRA), not a per-window budget — large budgets are effectively unreachable

Please select the area the issue is related to

Area/AIGateway (AI Gateway runtime/control plane), Area/Policies

Please select the aspect the issue is related to

Aspect/AI (AI/LLM integration, MCP, AI readiness)

Description

The LLM Cost-Based Rate Limit policy does not accumulate spend across requests. The first request after deployment reduces the remaining budget by its own cost; every subsequent request reports the identical remaining budget, regardless of how many calls are made or how much time passes. Because accumulated spend never grows, the configured budget is never reached and the policy never returns 429 — cost-based spend governance is entirely non-functional.

Two signals show the mechanism:

1. The window is reported as already expired on every request. The policy is configured with a 3600s window, but the ratelimit response header always carries t=0 (0 seconds to reset), and x-ratelimit-reset is always ~1 second after the request:

   ratelimit-policy: "llm_cost_quota";q=1000000000;w=3600     <-- 1 hour window configured
   ratelimit:         "llm_cost_quota";r=999974500;t=0        <-- t=0, resets immediately
   x-ratelimit-reset: 1785230977                              <-- request time + 1s
   

2. The quota result is never recorded. In gateway-runtime dynamic metadata the cost quota's result is null, so nothing is persisted between requests — each request is evaluated from a zero baseline, charges its own cost, and emits headers derived from that:

   llm_cost_scale_factor            = 1000000000
   provider_name                    = "mistral-test"
   ratelimit:keys:964fb30f          = {"llm_cost_quota": "POST|/mistral-test/*|*"}
   ratelimit:result:964fb30f        = null            <-- never recorded
   template_handle                  = "mistralai"
   

   For contrast, a total_tokens quota on the same gateway does produce a populated result with a real countdown (t=1392):

   "QuotaName":"total_tokens","Result":{"Requested":1,"Consumed":1,"Limit":111,"Remaining":109,...}
   

Steps to Reproduce

1. Create an LLM provider using the built-in Mistral template (mistralai) with a valid Mistral API key, and deploy it to a gateway.

2. On the provider, Guardrails tab → add LLM Cost-Based Rate Limit with a budget of 1 USD / 1h, then add LLM Cost after it in the policy list (per the documented ordering requirement). Redeploy.

3. Invoke the provider repeatedly. Space the calls out to show that time is not the factor (Mistral's own limit on mistral-large-latest is 4 req/min, so ~18s apart works):

   for i in 1 2 3; do
     curl -sk -o /dev/null -D /tmp/h.txt \
       -X POST "https://localhost:8443/mistral-test/v1/chat/completions" \
       -H 'Content-Type: application/json' -H "X-API-Key: $KEY" \
       -d '{"model":"mistral-large-latest","messages":[{"role":"user","content":"Say hello!"}]}'
     echo "call $i: $(grep -iE '^(x-ratelimit-cost-remaining-dollars|ratelimit):' /tmp/h.txt | tr -d '\r' | tr '\n' ' ')"
     sleep 18
   done
   

Each response reports usage of 6 prompt + 15 completion = 21 total tokens, so every call has an identical, non-zero cost.

Expected

Accumulated spend grows with each call. Remaining budget decreases monotonically — roughly $1.000000 → $0.999974 → $0.999949 → $0.999923 … — and once the budget is exhausted, further requests receive 429. t counts down toward the 3600s window boundary.

Actual

The remaining budget is charged once and then frozen. Observed across 8 calls spanning ~5 minutes (both back-to-back and spaced 18s apart):

call 1: x-ratelimit-cost-remaining-dollars: 0.999974   ratelimit: "llm_cost_quota";r=999974500;t=0
call 2: x-ratelimit-cost-remaining-dollars: 0.999974   ratelimit: "llm_cost_quota";r=999974500;t=0
call 3: x-ratelimit-cost-remaining-dollars: 0.999974   ratelimit: "llm_cost_quota";r=999974500;t=0
...
call 8: x-ratelimit-cost-remaining-dollars: 0.999974   ratelimit: "llm_cost_quota";r=999974500;t=0

x-ratelimit-remaining is likewise pinned at 999974500 internal units on every response. All calls return 200; no 429 ever occurs. The budget would never be reached no matter how much traffic is sent.

Notes

• This reproduces against the real https://api.mistral.ai upstream (no mock involved) using the built-in mistralai template, so it is not an artifact of a stubbed backend or a custom template.
• llm_cost_scale_factor = 1000000000 and the dollar/internal-unit headers are internally consistent ($1.00 → 1000000000 units; charge of 25500 units → $0.0000255), so the scaling and header rendering appear correct. The defect is that the charge is not persisted.
• Minor, separate observation (unverified, flagging only in case it is relevant): the charge of ~$0.0000255 for 6 prompt + 15 completion tokens on mistral-large-latest looks lower than published Mistral list pricing would suggest. This may simply reflect the rates in the bundled pricing database and is not the subject of this report.
• Possibly related to #2932 (provider-wide Token Count quota charges 1 unit per request instead of the token count) — both are rate-limit consumption-accounting defects in the same gateway version, though the failure modes differ: there the quota is recorded but mischarged, here it is not recorded at all.

Severity Level of the Issue

Severity/Critical (Core functionality is broken but there is a workaround. Need urgent attention)

Environment Details (with versions)

• AI Gateway 1.2.0-beta — gateway-controller reports version=1.2.0-beta git_commit=73e0a0a build_date=2026-07-22T09:17:35Z; gateway-runtime image ghcr.io/wso2/api-platform/gateway-runtime:1.2.0-beta
• AI Workspace 1.0.0-beta
• Platform API 0.13.0
• Bundled policies: llm-cost-based-ratelimit v1.1.0, llm-cost v1.0.3
• Docker Compose (standalone distributions), macOS (darwin 23.4.0)
• Provider template: mistralai; model mistral-large-latest; rate limit backend: memory (default)

Reproduce the issue with the built-in mistralai template and inspect gateway-runtime dynamic metadata for the llm_cost_quota result. Compare its null result and t=0 headers with the populated total_tokens quota result, then trace the LLM Cost-Based Rate Limit and LLM Cost policy interaction. Done means spend accumulates across requests, the remaining budget decreases, the window countdown is valid, and exhausted budgets return 429.