google / google/jaxopt

Implement a `split_rngs` mechanism, like in `flax.linen.scan`

Open
#432 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
76
Avg merge
2d 21h
Merged PRs (30d)
1

Description

I have the following problem: I want to optimize/find the fixed point of a function that uses flax's [`Dropout`](https://flax.readthedocs.io/en/latest/guides/dropout.html) and therefore under the hood it uses [`make_rng`](https://flax.readthedocs.io/en/latest/notebooks/flax_sharp_bits.html).
What this means, is that when the function is called multiple times inside the optimizer/fixed point solver, the dropout mask will change between calls.

For recurrent neural networks whose cells use dropout, this is fixed in flax using a `split_rngs` mechanism available in the [`scan` function](https://flax.readthedocs.io/en/latest/api_reference/_autosummary/flax.linen.scan.html).
Namely, the doc says:
> `split_rngs` – Split PRNG sequences will be different for each loop iterations. If split is False the PRNGs will be the same across iterations.

The same is available for the [while loop](https://flax.readthedocs.io/en/latest/api_reference/_autosummary/flax.linen.while_loop.html).

In my case, I would basically like to do something like `split_rngs={"dropout": False}`.

I think the best way to tackle this would be to implement special while loops for flax cases in [here](https://github.com/google/jaxopt/blob/main/jaxopt/_src/loop.py). Happy to discuss the API and whether this feature makes sense.

My basic use case is to implement Deep Equilibrium Models where the fixed point defining function uses dropout.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.