Cache hit/miss/eviction metrics in the Prometheus registry
@moonming is already working on this.
Since Jul 15, 2026.
- Dominant language
- Rust
- Stars
- 157
- Forks
- 32
- Avg merge
- 1h 25m
- Merged PRs (30d)
- 145
Description
Description
Cache policies support memory and Redis backends, TTL, scope by model or API key, a cache-status response header, and tokens-saved on the usage record. But there is no Prometheus counter for cache hits, misses, or evictions in the metrics registry — saved tokens are recorded on the usage event, so hit rate can be reconstructed in the analytics UI, but it cannot be scraped, alerted on, or graphed alongside the other aisix_* series.
Requested: hit / miss / eviction counters in the Prometheus registry, labelled consistently with the existing series (policy, model, backend).
Scope note: on-demand purge is deliberately not part of this issue — that is tracked in #89.
Why
Cache hit rate is the number that justifies the cache in a cost review, and it belongs in the same scrape and alerting path as every other metric rather than only in an analytics view. Operators cannot alert on "hit rate collapsed after a prompt-template change" today.
Priority
Medium.
Prior art
| Product | Has it | Reference |
|---|---|---|
| LiteLLM | Partial | litellm_cache_hits_metric / _misses_metric (in code, not yet in docs) |
| Kong AI Gateway | Partial | cache_status label on ai_llm_requests_total |
| Portkey | Partial | cacheStatus label on /metrics |
| Bifrost | Partial | bifrost_cache_hits_total (hits only) |
| Cloudflare AI Gateway | Partial | dashboard + logs, not Prometheus |
| Envoy AI Gateway | No | none found |
| TrueFoundry AI Gateway | Partial | dashboard "Cache Hit %" |
| Helicone | Partial | dashboard only |
Hit/miss telemetry is standard across peers; LiteLLM ships dedicated counters.
Surveyed 2026-07-15; every claim rests on a fetched docs/source page.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.