tensor parallel with uneven number of GPU's?
Open
- Dominant language
- C
- Stars
- 22.3k
- Forks
- 2.1k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 4
Description
> --cuda-tensor-parallel splits DeepSeek V4 Flash tensor and routed-expert work across an even number of GPUs.
Is this "even number of GPUs" limitation actually present in the ds4 code? Does it need to be? I know that vllm has such a limitation, but I don't believe llama-server does, for example.
It would be great to get tensor-parallel on (my) 5x3090 or maybe (someone else's) 3x32GB, for example :)
Contributor guide
Assessment
This issue has not been assessed yet.