invoke-ai / invoke-ai/invoke-training

Support for Gefen Optimizer

Open
#162 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
206
Forks
35
PR merge metrics
No merged PRs in 30d

Description

The newly released [Gefen](https://github.com/ndvbd/Gefen) optimizer is a drop-in replacement for AdamW that [claims](https://arxiv.org/pdf/2606.13894#page=10) an 8x memory reduction with comparable quality. It can be easily added as it is already compatible with Transformers, and has the potential to democratize training to a significant degree if it works as well as is being claimed. It may be preferable to trial the [Gefen-X fork](https://github.com/thad0ctor/Gefen-X) by @thad0ctor as the original author is deleting issues and appears disinterested in fixing up the code for modern architectures.

Contributor guide

No contributing guide indexed for this repository

Research direction

The payload names no repository files, tests, or entry points. Start by reviewing how Transformers optimizers are configured, then compare the Gefen and Gefen-X repositories for current-architecture compatibility. Done means Gefen support is integrated and usable in training with appropriate validation against the existing AdamW path.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.