invoke-ai / invoke-ai/invoke-training
Support for Gefen Optimizer
- Dominant language
- Python
- Stars
- 206
- Forks
- 35
- PR merge metrics
- No merged PRs in 30d
Description
The newly released [Gefen](https://github.com/ndvbd/Gefen) optimizer is a drop-in replacement for AdamW that [claims](https://arxiv.org/pdf/2606.13894#page=10) an 8x memory reduction with comparable quality. It can be easily added as it is already compatible with Transformers, and has the potential to democratize training to a significant degree if it works as well as is being claimed. It may be preferable to trial the [Gefen-X fork](https://github.com/thad0ctor/Gefen-X) by @thad0ctor as the original author is deleting issues and appears disinterested in fixing up the code for modern architectures.
Contributor guide
No contributing guide indexed for this repository
Research direction
The payload names no repository files, tests, or entry points. Start by reviewing how Transformers optimizers are configured, then compare the Gefen and Gefen-X repositories for current-architecture compatibility. Done means Gefen support is integrated and usable in training with appropriate validation against the existing AdamW path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100