[Feature] Prefix cache hit/miss/eviction statistics to detect cache thrashing
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
### Motivation
I'm trying to optimize a production scenario where I need to fit a 70B parameter model within 48GB of VRAM, and after the model weights there is only enough room for about 20 chat threads in the prefix cache. I'm trying to work out how much of a bottleneck this is in "the real world" - i.e. with actual traffic, since there are a lot of factors that are a bit difficult to test - different prompt lengths, bursts of requests, various amounts of overlap between different chat threads, etc.
It would be great if there were some way to count evictions so it would be clear if the limited VRAM for prefix cache is causing a bottleneck.
(Even more ideal would be if the cache manager kept hashes of recently evicted content so it can determine whether a recently evicted prefix was "requested", so it would be able to automatically detect "cache thrashing")
### Related resources
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.