EnzymeAD / EnzymeAD/Enzyme

Curious if there's a way to get gradients wrt final instead of initial values for mutating functions

Open
#2,330 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
LLVM
Stars
1.7k
Forks
188
Avg merge
1d 22h
Merged PRs (30d)
26

Description

More of something I'm wondering than an issue. I posted this on the mailing list but I'm not sure anyone sees that.

Consider the following example:
```
double f(double& x) {
x = x * x;
return x * x;
}
```
Here the gradient of `f` at `x = 1` wrt the initial value of `x` is 4, and the gradient wrt the final value of `x` is 2. Enzyme calculates the former, but what if I'm interested in the latter? I get the impression that this is doable but there's no api for this in Enzyme. What would it take to hack this on?

I'm still studying the internals of Enzyme and I'll probably find the answer to my question, but it would help me to hear how other people conceptualize this.

The motivation is to play with the idea of a neural network that can modify it's own weights. After the forward pass the initial weights are gone, so what we want to optimize is the final weights. Not sure if this is even worth trying, but I've been thinking about it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.