api7 / api7/aisix

Cache hit/miss/eviction metrics in the Prometheus registry

Open
#769 1 comment 1 reaction 1 assignee View on GitHub

@moonming is already working on this.

Since Jul 15, 2026.

enhancement
Dominant language
Rust
Stars
157
Forks
32
Avg merge
1h 25m
Merged PRs (30d)
145

Description

Description

Cache policies support memory and Redis backends, TTL, scope by model or API key, a cache-status response header, and tokens-saved on the usage record. But there is no Prometheus counter for cache hits, misses, or evictions in the metrics registry — saved tokens are recorded on the usage event, so hit rate can be reconstructed in the analytics UI, but it cannot be scraped, alerted on, or graphed alongside the other aisix_* series.

Requested: hit / miss / eviction counters in the Prometheus registry, labelled consistently with the existing series (policy, model, backend).

Scope note: on-demand purge is deliberately not part of this issue — that is tracked in #89.

Why

Cache hit rate is the number that justifies the cache in a cost review, and it belongs in the same scrape and alerting path as every other metric rather than only in an analytics view. Operators cannot alert on "hit rate collapsed after a prompt-template change" today.

Priority

Medium.


Prior art

Product Has it Reference
LiteLLM Partial litellm_cache_hits_metric / _misses_metric (in code, not yet in docs)
Kong AI Gateway Partial cache_status label on ai_llm_requests_total
Portkey Partial cacheStatus label on /metrics
Bifrost Partial bifrost_cache_hits_total (hits only)
Cloudflare AI Gateway Partial dashboard + logs, not Prometheus
Envoy AI Gateway No none found
TrueFoundry AI Gateway Partial dashboard "Cache Hit %"
Helicone Partial dashboard only

Hit/miss telemetry is standard across peers; LiteLLM ships dedicated counters.

Surveyed 2026-07-15; every claim rests on a fetched docs/source page.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.