NVIDIA-NeMo / NVIDIA-NeMo/Emerging-Optimizers

[Optimizer Proposal] NEAT: conflict-aware optimization via directional gradient correction

Open
#295 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
274
Forks
51
Avg merge
1d 3h
Merged PRs (30d)
6

Description

Is your feature request related to a problem?

Many emerging optimizers improve scaling, conditioning, curvature approximation, or optimizer memory.

A complementary problem is directional gradient conflict.

In some training regimes, the current gradient can directly oppose a stateful optimization signal such as momentum. Standard first-order optimizers generally continue accumulating that gradient without explicitly measuring the disagreement.

I would like to propose NEAT, Nash-Equilibrium Adaptive Training, as an experimental conflict-aware optimizer for Emerging-Optimizers.

Reference implementation:

https://github.com/ItCodinTime/NEAT

NEAT is also listed in the official Keras ecosystem:

https://keras.io/getting_started/ecosystem/

Proposed method

For current gradient g_t and opponent proxy o_t, NEAT measures conflict using negative cosine similarity.

conflict = relu(-cos(g_t, o_t))
projection = <g_t, o_t> / (||o_t||^2 + eps) * o_t
correction = -alpha * conflict * projection
g_corrected = g_t + correction

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no repository file, test, or entry point. Start by reviewing the proposed NEAT equations and the linked reference implementation, then inspect how Emerging-Optimizers defines and tests optimizers; done would require an agreed integration scope and validation criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.