google-deepmind / google-deepmind/asyncdiloco
Why is Async Local SGD better even without momentum
Open
- Dominant language
- Jupyter Notebook
- Stars
- 51
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
I wondered why, in the experiment where momentum was turned off, the async local sgd still outperformed vanilla local-sgd. Shouldn't the gradient staleness significantly impact wall clock training time, which is why async-local sgd suggests a delayed momentum update?
Contributor guide
Assessment
This issue has not been assessed yet.