deepspeedai / deepspeedai/DeepSpeed
llama2-13b-tp8 long sentence input error.[BUG]
@HeyangQin is already working on this.
Since Dec 22, 2023.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
import mii
path = "path/to/llama2_13b_huggingface_path"
pipe = mii.pipeline(path, max_length=133120)
prompt = "test "*10000
response = pipe(prompt, max_new_tokens=512, max_length=133120, do_sample=False)
python3.9/site-packages/deepspeed/inference/v2/ragged/sequence_descriptor.py", line 258, in extend_kv_cache
shadow_alloc_group[cur_blocks:cur_blocks + new_blocks].copy_(new_group_ids)
RuntimeError: The size of tensor a (8) must match the size of tensor b (12) at non-singleton dimension 0
I gebug the code find:
len(shadow_alloc_group) = 32
len(new_group_ids) = 12
cur_blocks = 24
new_blocks = 12
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.