bigscience-workshop / bigscience-workshop/metadata
feat: save the model and stop training based on `exit-duration-in-mins`
Open
- Dominant language
- Python
- Stars
- 29
- Forks
- 11
- PR merge metrics
- No merged PRs in 30d
Description
As discussed a long time ago in a meeting it would be really great if we had a feature to save the model and stop training after a certain time as the jobs on the JZ cluster are limited to 20 hours.
For example, in the architecture and scaling working group, they added the `exit-duration-in-mins` argument the library used to run trainings [`Megatron-DeepSpeed`](https://github.com/bigscience-workshop/Megatron-DeepSpeed/)
---
related: #37 (#42)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.