tensor parallel with uneven number of GPU's?
未关闭
- 主要语言
- C
- 星标
- 22.3k
- 派生
- 2.1k
- 平均合并
- 1 天 3 小时
- 30 天内合并 PR
- 4
描述
> --cuda-tensor-parallel splits DeepSeek V4 Flash tensor and routed-expert work across an even number of GPUs.
Is this "even number of GPUs" limitation actually present in the ds4 code? Does it need to be? I know that vllm has such a limitation, but I don't believe llama-server does, for example.
It would be great to get tensor-parallel on (my) 5x3090 or maybe (someone else's) 3x32GB, for example :)
贡献指南
评估
这个 Issue 还没有评估数据。