InternLM / InternLM/lmdeploy

[Feature] Prefix cache hit/miss/eviction statistics to detect cache thrashing

Open
#1,942 5 comments 0 reactions 2 assignees Claimed by @zhyncs View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### Motivation

I'm trying to optimize a production scenario where I need to fit a 70B parameter model within 48GB of VRAM, and after the model weights there is only enough room for about 20 chat threads in the prefix cache. I'm trying to work out how much of a bottleneck this is in "the real world" - i.e. with actual traffic, since there are a lot of factors that are a bit difficult to test - different prompt lengths, bursts of requests, various amounts of overlap between different chat threads, etc.

It would be great if there were some way to count evictions so it would be clear if the limited VRAM for prefix cache is causing a bottleneck.

(Even more ideal would be if the cache manager kept hashes of recently evicted content so it can determine whether a recently evicted prefix was "requested", so it would be able to automatically detect "cache thrashing")

### Related resources

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.