NVIDIA / NVIDIA/TensorRT-LLM

[Feature]: Extend the v1 KV cache manager with a disk (L3) tier and per-request retention

Open
#16,251 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

KV-Cache Management
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

🚀 The feature, motivation and pitch

The v1 KV cache block manager currently supports two tiers — GPU (primary) and host
(secondary). We'd like to extend it with a third, disk-backed tier: blocks evicted from
the host tier spill to local disk and are reused on a later prefix hit instead of recomputed,
extending KV reuse well beyond DRAM.

We'd also like to add a per-request retention period — a request can ask that its KV cache
stay reusable for a given amount of time. The engine has no such control today.

We've implemented both for the v1 manager and validated them under production traffic at
DeepInfra. A PR is ready and will be opened shortly.

Alternatives

KVCacheManagerv2 does have a disk tier, but in our production testing at DeepInfra it
cannot sustain production traffic (it has to run with the overlap scheduler disabled, and
even then it can't keep up), and it has no per-request retention. So it does not cover the
default v1 path, nor the retention capability we need.

Additional context

Implementation (C++ + Python, with a disk-tier unit-test suite and benchmarks) is ready; we
will open the PR shortly.


  • I searched existing/past issues and checked the documentation and examples.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the v1 KV cache manager entry points in the C++ and Python implementation, then review the disk-tier unit-test suite and benchmarks mentioned in the issue. Done means the v1 manager supports GPU, host, and disk tiers plus per-request retention, with the accompanying tests and benchmark coverage passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
ai, backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.