deepspeedai / deepspeedai/DeepSpeed
[REQUEST] Deepspeed Inference Supports VL (vision language) model
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Is your feature request related to a problem? Please describe.
We have been using deepspeed.init_inference API for speeding up inference for text only models (e.g. mistral, qwen 2.5 series) with success. Was hoping we can extend support for vision language models as well, e.g. qwen 2 vl, etc, which is currently not supported.
Describe the solution you'd like
deepspeed.init_inferenceto work for vision language models (for both embedding use case as well as generation use case)- and also make extending with our own model's tutorial clearer/cleaner.
Describe alternatives you've considered
N/A
Additional context
N/A
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the deepspeed.init_inference API and its existing text-only model support. Compare the requirements for vision-language embedding and generation use cases, then review the tutorial guidance for extending support to custom models. Done means both requested use cases work for vision-language models and the extension tutorial is clearer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100