NVIDIA / NVIDIA/TensorRT-LLM

[Bug]: Scaffolding KV cache hints use an unregistered HTTP endpoint

Open Beginner friendly
#19,281 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

KV-Cache Management
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

System info

Source inspected at main commit 7b1bedbbb. This is a client/server HTTP route mismatch; no GPU inference run was performed.

Problem

With KV cache hints enabled, OpenaiWorker posts message-based DropKVCacheTask requests to base_url + "kv_cache_hints". For the usual OpenAI base URL http://localhost:8000/v1/, this targets /v1/kv_cache_hints.

The TRT-LLM resource governor instead registers POST /_control/kv_cache/truncate (and the compatibility alias /_resource_governor/truncate). It does not register /v1/kv_cache_hints, so the request cannot reach the truncate handler and receives 404 on a directly connected server.

Reproduction

Configure OpenaiWorker with kv_cache_hint_enabled=True and an OpenAI client base URL ending in /v1/, then execute a DropKVCacheTask. Inspect the outgoing POST path and compare it with the routes in tensorrt_llm/serve/resource_governor.py.

Expected behavior / proposed fix

Post to /_control/kv_cache/truncate, stripping the optional trailing /v1 and normalizing the trailing slash. Preserve the existing request body and Authorization header.

The token-based truncate method already uses the sibling control-plane endpoint. The message-based path changed to kv_cache_hints in #14397; before that it used the registered resource-governor endpoint.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the message-based DropKVCacheTask path in OpenaiWorker and compare its outgoing URL with the routes in tensorrt_llm/serve/resource_governor.py. Reproduce with kv_cache_hint_enabled=True and an OpenAI base URL ending in /v1/, then verify that the request reaches /_control/kv_cache/truncate with the existing body and Authorization header.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.