agentscope-ai / agentscope-ai/Trinity-RFT

[FEATURE] Support for Tensor-Parallel

未關閉
#285 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
Python
星號
701
分支
79
平均合併
8 小時 7 分鐘
30 天內合併 PR
1

描述

If I understand correctly, for more precise set-up of TP within FSDP2 framework, `parallelize_module(...)` must be called explicitly with a `tp_plan` (https://docs.pytorch.org/tutorials/intermediate/TP_tutorial.html#combine-tensor-parallel-with-fully-sharded-data-parallel-together), example at https://github.com/pytorch/torchtune/blob/main/recipes/full_finetune_distributed.py

But as I found, Verl/Trinity do not prepare any explicit TP-plan and do not invoke `parallelize_module(...)`. Am I correct that TP is not currently supported under the FSDP/2 trainer?

Thanks!

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。