[Feature]: Extend the v1 KV cache manager with a disk (L3) tier and per-request retention
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
🚀 The feature, motivation and pitch
The v1 KV cache block manager currently supports two tiers — GPU (primary) and host
(secondary). We'd like to extend it with a third, disk-backed tier: blocks evicted from
the host tier spill to local disk and are reused on a later prefix hit instead of recomputed,
extending KV reuse well beyond DRAM.
We'd also like to add a per-request retention period — a request can ask that its KV cache
stay reusable for a given amount of time. The engine has no such control today.
We've implemented both for the v1 manager and validated them under production traffic at
DeepInfra. A PR is ready and will be opened shortly.
Alternatives
KVCacheManagerv2 does have a disk tier, but in our production testing at DeepInfra it
cannot sustain production traffic (it has to run with the overlap scheduler disabled, and
even then it can't keep up), and it has no per-request retention. So it does not cover the
default v1 path, nor the retention capability we need.
Additional context
Implementation (C++ + Python, with a disk-tier unit-test suite and benchmarks) is ready; we
will open the PR shortly.
- I searched existing/past issues and checked the documentation and examples.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the v1 KV cache manager entry points in the C++ and Python implementation, then review the disk-tier unit-test suite and benchmarks mentioned in the issue. Done means the v1 manager supports GPU, host, and disk tiers plus per-request retention, with the accompanying tests and benchmark coverage passing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- ai, backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100