f-dangel / f-dangel/backpack

[Feature Request] Levenberg Marquardt

Open
#293 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
617
Forks
57
PR merge metrics
No merged PRs in 30d

Description

# Feature

Implementation of Levenberg Marquardt optimizer.

# Motivation

In MATLAB, the `feedforwardnet` supports many optimizers, and some are not implemented in PyTorch, such as `trainlm` ([Levenberg Marquardt](https://www.mathworks.com/help/deeplearning/ref/trainlm.html)). From tests from our gourp and others, Levenberg Marquardt performs much better and needs less epochs to coverage than first-order optimizers (Adam, SGD with Momentum) on regression model. Some issues in PyTorch issue list hope for a implementation of Levenverg Marquardt algorithm, but there are no more reponse. It will be helpful to implement the Levenberg Marquardt optimizer and I am willing to help test.

# Addition

Here are some helpful information and codes.

- An introduction on Wikipedia. [Levenberg–Marquardt algorithm](https://en.m.wikipedia.org/wiki/Levenberg–Marquardt_algorithm)
- An implementation for Tensorflow. [tf-levenberg-marquardt](https://github.com/fabiodimarco/tf-levenberg-marquardt)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.