Multiple gpus error vicuna-7b-v1.3: RuntimeError: probability tensor contains either inf, nan or element < 0
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
This code below is return an error, running in multi GPUs.
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.3 --num-gpus 3 --max-gpu-memory 6GiB
USER: what can you do?
ASSISTANT: пеamaire favor Arab Dra benep) Rev1 Storage78ITE마zigate point fail4check therefore ':-erserserszeroispecies Theirulleselfoproreu'> definitionSsm Traceback (most recent call last):
File "/FastChat/fastchat/serve/inference.py", line 174, in generate_stream
indices = torch.multinomial(probs, num_samples=2)
RuntimeError: probability tensor contains either inf, nan or element < 0
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported command with lmsys/vicuna-7b-v1.3 across three GPUs, then inspect fastchat/serve/inference.py around generate_stream line 174 and the values passed to torch.multinomial. Confirm the cause of the invalid probability tensor and verify that multi-GPU generation completes without this RuntimeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100