patrick-kidger / patrick-kidger/diffrax
RecursiveCheckpointAdjoint not working for two-level minimisation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 189
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 1
Description
Hi all, I've edited the introductory Neural ODE example to highlight a problem I'm facing with two-level optimisation: first (outer) level wrt the model, and second (inner) level wrt a parameter alpha. JAX throws a JaxStackTraceBeforeTransformation error if I use RecursiveCheckpointAdjoint, but everything runs if I use DirectAdjoint instead. In line with the recommendations in the documentation, I'd love to use the former adjoint rule. Please help, Thanks.
import equinox as eqx
import diffrax
import jax
import jax.numpy as jnp
data_size=2
class Func(eqx.Module):
mlp: eqx.nn.MLP
def __init__(self):
self.mlp = eqx.nn.MLP(
in_size=data_size+1,
out_size=data_size,
width_size=4,
depth=2,
activation=jax.nn.softplus,
key=jax.random.PRNGKey(0),
)
def __call__(self, t, y, args):
alpha = args[0]
y = jnp.concatenate([y, alpha])
return self.mlp(y)
class NeuralODE(eqx.Module):
func: Func
def __init__(self):
self.func = Func()
def __call__(self, ts, y0, alpha):
solution = diffrax.diffeqsolve(
diffrax.ODETerm(self.func),
diffrax.Tsit5(),
t0=ts[0],
t1=ts[-1],
dt0=ts[1] - ts[0],
y0=y0,
args=(alpha,),
# adjoint=diffrax.DirectAdjoint(), ## works fine ! 🎉
adjoint=diffrax.RecursiveCheckpointAdjoint(), ## throws a JaxStackTraceBeforeTransformation 😢
stepsize_controller=diffrax.PIDController(rtol=1e-3, atol=1e-6),
saveat=diffrax.SaveAt(ts=ts),
)
return solution.ys
def loss_fn(model, alpha):
ts = jnp.linspace(0, 1, 100)
y0 = jnp.zeros(data_size)
return jnp.mean(model(ts, y0, alpha) ** 2)
def inner_step(model, alpha):
alpha_grad = eqx.filter_grad(lambda alpha, model: loss_fn(model, alpha))(alpha, model)
return jnp.mean(alpha_grad)
def outer_step(model, alpha):
model_grad = eqx.filter_grad(inner_step)(model, alpha)
return model_grad
model = NeuralODE()
alpha = jnp.array([1.])
## Run the outer step
outer_step(model, alpha)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the introductory Neural ODE example and tracing the call from outer_step through inner_step, loss_fn, and diffrax.diffeqsolve. Compare RecursiveCheckpointAdjoint with DirectAdjoint and confirm that the two-level minimisation completes without the JaxStackTraceBeforeTransformation error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100