benchopt / benchopt/benchmark_nanogpt
TODO investigate ZerocopyOptimizer
Open
- Dominant language
- Python
- Stars
- 5
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
- It should be straight forward for AdamW where the udpates are coordinate wise so sharding works as intended.
- This is trickier for SOAP/Scion/Muon which needs to keep the matrices intact.
- look at this [implementation](https://github.com/SDLAML/disco).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the ZerocopyOptimizer TODO and the linked disco implementation. Compare the intended sharding behavior for AdamW with the matrix-preserving requirements of SOAP, Scion, and Muon. Done means the compatibility requirements are established and the optimizer behavior is verified for each named case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100