NVIDIA / NVIDIA/TensorRT-Edge-LLM
Does the current implementation support prefix/encoder cache?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 563
- Forks
- 135
- Avg merge
- 14h 13m
- Merged PRs (30d)
- 1
Description
I tested the current implementation and it seems that prefix cache is not yet supported. When I input the same image and prompt multiple times, the inference speed remains the same each time — neither the image nor the prompt appears to be reused.
Do you have plans to add prefix/encoder cache functionality (similar to what vLLM offers) in the future?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue does not name any implementation files, tests, or entry points. Begin by locating the inference path and any image, prompt, encoder, or prefix reuse mechanisms in the repository, then determine the expected cache behavior and how repeated inputs should be measured. Done means a defined cache design is implemented and verified against repeated image-and-prompt inference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100