facebookresearch / facebookresearch/fairscale

Support BF16 for FSDP

Open
#963 8 comments 3 reactions 1 assignee Claimed by @anj-s View on GitHub
FSDP
Dominant language
Python
Stars
3.4k
Forks
293
PR merge metrics
No merged PRs in 30d

Description

## Feature Request
Please support BF16 mixed-precision

## Additional context
Training with BF16 is usually more stable than fp16, which is very important when we want to train large models. Additionally, many models (e.g. T5) are trained with BF16 and if we want to continue training them with mixed-precision, using fp16 will result in NaNs.

Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.