Lightning-AI / Lightning-AI/LitServe
avoid sending content past the last token during batching / batched streaming
Open
Nobody has claimed this yet.
won't fix
- Dominant language
- Python
- Stars
- 3.9k
- Forks
- 304
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 6
Description
Find a way to avoid sending a lot of tokens past the last token for a particular item in the batch (i.e. we need to trim past the EOS in encode_response, let's open an issue and create an example about it)
as a more immediate improvement, we need to find a way to avoid sending a lot of tokens past the last token for a particular item in the batch (i.e. we need to trim past the EOS in `encode_response`, let's open an issue and create an example about it)
Originally posted by @lantiga in https://github.com/Lightning-AI/litserve/pull/55#discussion_r1575203899
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.