Support left padding to forward batch prompts in a single step
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 4.7k
- Forks
- 536
- Avg merge
- 12h 12m
- Merged PRs (30d)
- 4
Description
Batches of variable-length prompts are currently not forwarded in a single step into the decoder. Only the tokens up to the minimum length in the batch are forwarded at once, and the remaining tokens are force-decoded in the decoding loop.
It could be more efficient to forward the batch in a single step but this requires supporting padding positions on the left. It means each entry in the batch has an offset in position-aware modules:
- softmax
- position encodings
- rotary embeddings
- padder
- MHA values "mask"
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the decoder's batch-forward path and the decoding loop described in the issue. Check how offsets are represented in softmax, position encodings, rotary embeddings, the padder, and MHA values masking; done means variable-length batches can use left padding and be forwarded in one step across all listed components.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100