kohya-ss / kohya-ss/sd-scripts
Support GaLore Optimazer
Open
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
The optimizer is memory efficient.
We can pretrain mistral-7B with 24GB.
https://github.com/jiaweizzhao/GaLore
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the linked GaLore project and examining this repository's existing training and optimizer entry points. Done means adding a defined GaLore integration that supports the stated memory-efficient pretraining goal for mistral-7B, with validation of the resulting training behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100