tensor parallel with uneven number of GPU's?
Abierto
- Lenguaje dominante
- C
- Estrellas
- 22.3k
- Forks
- 2.1k
- Merge medio
- 1 d 3 h
- PR fusionados (30 d)
- 4
Descripción
> --cuda-tensor-parallel splits DeepSeek V4 Flash tensor and routed-expert work across an even number of GPUs.
Is this "even number of GPUs" limitation actually present in the ds4 code? Does it need to be? I know that vllm has such a limitation, but I don't believe llama-server does, for example.
It would be great to get tensor-parallel on (my) 5x3090 or maybe (someone else's) 3x32GB, for example :)
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.