NVIDIA-NeMo / NVIDIA-NeMo/Emerging-Optimizers
[Optimizer Proposal] NEAT: conflict-aware optimization via directional gradient correction
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 274
- Forks
- 51
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 6
Description
Is your feature request related to a problem?
Many emerging optimizers improve scaling, conditioning, curvature approximation, or optimizer memory.
A complementary problem is directional gradient conflict.
In some training regimes, the current gradient can directly oppose a stateful optimization signal such as momentum. Standard first-order optimizers generally continue accumulating that gradient without explicitly measuring the disagreement.
I would like to propose NEAT, Nash-Equilibrium Adaptive Training, as an experimental conflict-aware optimizer for Emerging-Optimizers.
Reference implementation:
https://github.com/ItCodinTime/NEAT
NEAT is also listed in the official Keras ecosystem:
https://keras.io/getting_started/ecosystem/
Proposed method
For current gradient g_t and opponent proxy o_t, NEAT measures conflict using negative cosine similarity.
conflict = relu(-cos(g_t, o_t))
projection = <g_t, o_t> / (||o_t||^2 + eps) * o_t
correction = -alpha * conflict * projection
g_corrected = g_t + correction
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no repository file, test, or entry point. Start by reviewing the proposed NEAT equations and the linked reference implementation, then inspect how Emerging-Optimizers defines and tests optimizers; done would require an agreed integration scope and validation criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100