google-deepmind / google-deepmind/optax

Problems when jitting Adafactor with inject_hyperparams.

Open
#412 0 comments 2 reactions 1 assignee Claimed by @mkunesch View on GitHub
type:bug
Dominant language
Python
Stars
2.3k
Forks
369
Avg merge
10h 15m
Merged PRs (30d)
7

Description

When wrapping `optax.adafactor` with `optax.inject_hyperparams` without specifying `static_args`

```
optax.inject_hyperparams(optax.adafactor)(learning_rate=0.1)
```

the `init` function of the resulting `GradientTransformation` cannot be jit compiled. The reason is that by default `inject_hyperparams` treats all arguments as dynamic and one of the argument has to be static to avoid a `TracerError`. A workaround is to specify the static argument:

```
optax.inject_hyperparams(optax.adafactor, static_args=("min_dim_size_to_factor",))(learning_rate=0.1)
```

However, this is not ideal since it requires the user to know which arguments should be static and which ones can be dynamic.

We should:

- Add a test to check whether any other optimizers are affected.
- Change the implementations so that all optimizers wrapped in inject_hyperparams can be jit compiled without any arguments being specified as static.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.