[Bug]: Scaffolding KV cache hints use an unregistered HTTP endpoint
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
System info
Source inspected at main commit 7b1bedbbb. This is a client/server HTTP route mismatch; no GPU inference run was performed.
Problem
With KV cache hints enabled, OpenaiWorker posts message-based DropKVCacheTask requests to base_url + "kv_cache_hints". For the usual OpenAI base URL http://localhost:8000/v1/, this targets /v1/kv_cache_hints.
The TRT-LLM resource governor instead registers POST /_control/kv_cache/truncate (and the compatibility alias /_resource_governor/truncate). It does not register /v1/kv_cache_hints, so the request cannot reach the truncate handler and receives 404 on a directly connected server.
Reproduction
Configure OpenaiWorker with kv_cache_hint_enabled=True and an OpenAI client base URL ending in /v1/, then execute a DropKVCacheTask. Inspect the outgoing POST path and compare it with the routes in tensorrt_llm/serve/resource_governor.py.
Expected behavior / proposed fix
Post to /_control/kv_cache/truncate, stripping the optional trailing /v1 and normalizing the trailing slash. Preserve the existing request body and Authorization header.
The token-based truncate method already uses the sibling control-plane endpoint. The message-based path changed to kv_cache_hints in #14397; before that it used the registered resource-governor endpoint.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the message-based DropKVCacheTask path in OpenaiWorker and compare its outgoing URL with the routes in tensorrt_llm/serve/resource_governor.py. Reproduce with kv_cache_hint_enabled=True and an OpenAI base URL ending in /v1/, then verify that the request reaches /_control/kv_cache/truncate with the existing body and Authorization header.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100