tensor parallel with uneven number of GPU's?
Ouverte
- Langage dominant
- C
- Étoiles
- 22.3k
- Forks
- 2.1k
- Merge moyen
- 1 j 3 h
- PR mergées (30 j)
- 4
Description
> --cuda-tensor-parallel splits DeepSeek V4 Flash tensor and routed-expert work across an even number of GPUs.
Is this "even number of GPUs" limitation actually present in the ds4 code? Does it need to be? I know that vllm has such a limitation, but I don't believe llama-server does, for example.
It would be great to get tensor-parallel on (my) 5x3090 or maybe (someone else's) 3x32GB, for example :)
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.