agentscope-ai / agentscope-ai/Trinity-RFT
[FEATURE] Support for Tensor-Parallel
未关闭
enhancement
- 主要语言
- Python
- 星标
- 701
- 派生
- 79
- 平均合并
- 8 小时 7 分钟
- 30 天内合并 PR
- 1
描述
If I understand correctly, for more precise set-up of TP within FSDP2 framework, `parallelize_module(...)` must be called explicitly with a `tp_plan` (https://docs.pytorch.org/tutorials/intermediate/TP_tutorial.html#combine-tensor-parallel-with-fully-sharded-data-parallel-together), example at https://github.com/pytorch/torchtune/blob/main/recipes/full_finetune_distributed.py
But as I found, Verl/Trinity do not prepare any explicit TP-plan and do not invoke `parallelize_module(...)`. Am I correct that TP is not currently supported under the FSDP/2 trainer?
Thanks!
贡献指南
评估
这个 Issue 还没有评估数据。