deepspeedai / deepspeedai/DeepSpeed
[REQUEST] Make DeepSpeed FastGen support tokens as input?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Is your feature request related to a problem? Please describe.
Current examples and documents I found can only support text as input:
import mii
pipe = mii.pipeline("mistralai/Mistral-7B-v0.1")
response = pipe(["DeepSpeed is", "Seattle is"], max_new_tokens=128)
print(response)
For example, If I want to use the chat function in mistralai/Mixtral-8x7B-Instruct-v0.1, theoretically, I have to make the following formats
Describe the solution you'd like
If I can only use text as input, I cannot really do that.
Describe alternatives you've considered
VLLM supports tokens as input
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the MII pipeline example in the issue and compare the requested token-input behavior with the vLLM alternative mentioned. Done means DeepSpeed FastGen accepts token inputs for inference, including the chat use case described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100