Lightning-AI / Lightning-AI/litgpt
Add support for memory-efficient and faster optimizers
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.5k
- Avg merge
- 15h 37m
- Merged PRs (30d)
- 1
Description
Maybe GaLore (#1192) should be changed from `GaloreArgs` to `OptimizerArgs` after all. Then we can also more easily consider other variants such as BAdam (BAdam: A Memory Efficient Full Parameter Training Method for Large Language Models, https://arxiv.org/abs/2404.02827).
The experiments from [here](https://github.com/Ledzy/BAdam) look very compelling. And it only adds 1 hyperparameter:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing GaLore work in issue #1192 and the linked BAdam paper and experiments. Compare the proposed OptimizerArgs approach with the current GaloreArgs design, then define what support for BAdam and other memory-efficient optimizer variants would need to cover. Done means the requested optimizer support and its configuration path are clearly specified and validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100