deepspeedai / deepspeedai/DeepSpeed
[BUG] HybridEngine llama2 70B generate result is wrong and "The size of tensor a (12) must match the size of tensor b (48) at non-singleton dimension 0" when inference_tp_size > 1
Open
Nobody has claimed this yet.
bug
compression
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
HybridEngine llama2 70B generate have two bugs:
- When inference_tp_size == 1, generate result is not right.
- When inference_tp_size > 1,
File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 593, in _prepare_decoder_attention_mask
expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
The size of tensor a (12) must match the size of tensor b (48) at non-singleton dimension 0
System info (please complete the following information):
- transformers: 0.10.3
- deepspeed: 4.31.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing HybridEngine llama2 70B generation with inference_tp_size set to 1 and greater than 1, using the reported transformers 0.10.3 and deepspeed 4.31.0 versions. Inspect transformers/models/llama/modeling_llama.py at _prepare_decoder_attention_mask and compare the generated result and tensor shapes; done means correct generation in both configurations without the size-mismatch error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100