facebookresearch / facebookresearch/fairscale
[FSDP] fp32_reduce_scatter can be decoupled from mixed precision
Open
FSDP
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 293
- PR merge metrics
- No merged PRs in 30d
Description
while working on this pr: https://github.com/facebookresearch/fairscale/pull/612
It seems that both `fp32_reduce_scatter` and perhaps even `cpu_offload` options can be made decoupled from mixed precision model. That could be useful in different use cases?
cc: @myleott
Contributor guide
Assessment
This issue has not been assessed yet.