agentscope-ai / agentscope-ai/Trinity-RFT

[FEATURE] Support for Tensor-Parallel

Abierto
#285 1 comentario 0 reacciones 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
Python
Estrellas
701
Forks
79
Merge medio
8 h 7 min
PR fusionados (30 d)
1

Descripción

If I understand correctly, for more precise set-up of TP within FSDP2 framework, `parallelize_module(...)` must be called explicitly with a `tp_plan` (https://docs.pytorch.org/tutorials/intermediate/TP_tutorial.html#combine-tensor-parallel-with-fully-sharded-data-parallel-together), example at https://github.com/pytorch/torchtune/blob/main/recipes/full_finetune_distributed.py

But as I found, Verl/Trinity do not prepare any explicit TP-plan and do not invoke `parallelize_module(...)`. Am I correct that TP is not currently supported under the FSDP/2 trainer?

Thanks!

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.