bigscience-workshop / bigscience-workshop/bigscience
Why is deepspeed enabled in the Bloom training script?
Open
- Dominant language
- Shell
- Stars
- 1k
- Forks
- 102
- PR merge metrics
- No merged PRs in 30d
Description
Why is the value of Zero-State 0 when deepspeed is enabled in the Bloom training script? Can the Bloom model be trained and the loss curve is aligned when deepspeed is disabled? Thanks very much.
```shell
DEEPSPEED_ARGS=" \
--deepspeed \
--deepspeed_config ${config_json} \
--zero-stage ${ZERO_STAGE} \
--deepspeed-activation-checkpointing \
"
```
Contributor guide
Assessment
This issue has not been assessed yet.