Use dynamic shapes for kv-cache
Open
@CharlieL7 is already working on this.
Since Oct 24, 2024.
enhancement
- Dominant language
- C++
- Stars
- 333
- Forks
- 150
- Avg merge
- 4d 19h
- Merged PRs (30d)
- 54
Description
There is two parts for this:
- Use dynamic shapes to handle the past sequence length.
- Similar to dynamic batching, we can make two different submodule for each GQA operator for max sequence length and sequence length of 1. Then use a select operator to select the different module based on the inputs sizes(perhaps its possible to reuse the
selectfrom dynamic batching).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.