训练内存占用问题

您好，我在使用4张卡，256G内存运行微调代码时会因为多个rank都分配初始化空间产生内存溢出问题。而使用两张卡，只开启两个rank则可以成功加载模型。
而在FSDP加载模型后，内存占用会掉回约模型的4倍大小，请问是否有方法可以减少每个rank对内存的临时占用，或者让它们加载顺序进行？
感谢您拨冗，期待您的回复。

Start by reproducing the fine-tuning run with two and four cards, comparing peak host-memory use across ranks during FSDP model loading. Inspect the repository's FSDP loading and initialization entry points; the issue is done when the four-card configuration no longer exhausts 256G of memory, with the resulting memory behavior documented.

Written by the indexing model from the issue text.