huggingface / huggingface/transformers
[Deepspeed zero3] lazy weights init
Open
DeepSpeed
WIP
- Dominant language
- Python
- Stars
- 166k
- Forks
- 34.6k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 276
Description
I'm pretty sure we need to follow up to the lazy weights init feature https://github.com/huggingface/transformers/pull/11471
and add under zero3 `deepspeed.zero.GatheredParameters` here (or inside `_init_weights`):
https://github.com/huggingface/transformers/pull/11471/files#diff-6b72b98c4c2dcfc6cc606843917733f5d858374fbc22a735ff483bbc0c1e63eaR1275-R1276
plus need a test.
Contributor guide
Assessment
This issue has not been assessed yet.