deepspeedai / deepspeedai/DeepSpeed
[BUG] DeepSpeed Hybrid Engine Does not Work for Mistral-7B
Open
Nobody has claimed this yet.
bug
deepspeed-chat
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
For mistral-7b models, it seem that hybrid engine does not work in generation. It can be seen the generation time reported below:
|E2E latency=84.05s |Gather latency=0.00s (0.00%) |Generate time=0.00s (0.00%) |Training time=17.70s (21.06%) |Others=66.34 (78.94%)|CurSamplesPerSec=1.14 |AvgSamplesPerSec=2.28
After investigation, the following codes do not work
https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/runtime/hybrid_engine.py#L357-L359
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with deepspeed/runtime/hybrid_engine.py at lines 357-359 and reproduce generation with a Mistral-7B model. Trace why the hybrid engine does not perform generation there, then verify that generation completes and the reported Generate time is no longer 0.00s.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100