[Performance]: Can Scaffolding give more information about kvcache-reuse to TensorRT-LLM?
Open
@dc3671 is already working on this.
Since Aug 22, 2025.
KV-Cache Management
Performance
Scaffolding
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
Proposal to improve performance
This is a discussion.
Scaffolding can allow users to explicitly mark the relationships between different inference requests in an agent application request. Will this be helpful for the management of kvcache? If so, how should we utilize this information?
Report of performance regression
No response
Misc discussion on performance
No response
Your current environment (if you think it is necessary)
System Information:
- OS:
- Python version:
- CUDA version:
- GPU model(s):
- Driver version:
- TensorRT version:
- PyTorch version:
- TensorRT-LLM version:
Detailed output:
Paste the output of the above commands here
Before submitting a new issue...
- Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.