google / google/flax

Dropout + `nn.jit`

Open
#3,171 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
7.3k
Forks
833
Avg merge
5h 11m
Merged PRs (30d)
5

Description

Hi there,

I was following this guide, [flax.linen.Dropout](https://flax.readthedocs.io/en/latest/guides/dropout.html#more-flax-examples-that-use-module-make-rng). Then I decided to add `nn.jit`, and started getting `MyModel.__call__() missing 1 required positional argument: 'training'`, even though I passed it.

### System information
- OS Platform: macOS 12.4
- Flax, jax, jaxlib versions:
```
Name: flax
Version: 0.6.11
---
Name: jax
Version: 0.4.13
---
Name: jaxlib
Version: 0.4.13

```
- Python version `python3.10`
- No hardware acceleration was used

### Problem you have encountered:
```
Traceback (most recent call last):
File "/Users/artemsereda/Documents/IdeaProjects/kaggle-bliss/bug_report.py", line 34, in
main()
File "/Users/artemsereda/Documents/IdeaProjects/kaggle-bliss/bug_report.py", line 25, in main
variables = my_model.init(params_key, x, training=False)
TypeError: MyModel.__call__() missing 1 required positional argument: 'training'
```

### What you expected to happen:
I expect jitted and non-jitted version to work the same. Or am I missing something?

### Steps to reproduce:
```python
import jax
import jax.numpy as jnp
import flax.linen as nn

class MyModel(nn.Module):
num_neurons: int

@nn.compact
def __call__(self, x, training: bool):
x = nn.Dense(self.num_neurons)(x)
x = nn.Dropout(rate=0.5, deterministic=not training)(x)
return x

def main():
root_key = jax.random.PRNGKey(seed=0)
main_key, params_key, dropout_key = jax.random.split(key=root_key, num=3)
my_model = nn.jit(MyModel)(num_neurons=3)
x = jnp.empty((3, 4, 4))
variables = my_model.init(params_key, x, training=False)

if __name__ == "__main__":
main()
```

### Question

I have saw the warning
```
/Users/artemsereda/miniconda3/envs/py310/lib/python3.10/site-packages/flax/core/lift.py:111: RuntimeWarning: kwargs are not supported in jit, so "training" is(are) ignored
warnings.warn(msg.format(name, ', '.join(kwargs.keys())), RuntimeWarning)
```
so I decided to change my code to
```python
variables = my_model.init(params_key, x, False)
```
which then resulted in
```
Traceback (most recent call last):
File "/Users/artemsereda/Documents/IdeaProjects/kaggle-bliss/bug_report.py", line 25, in
main()
File "/Users/artemsereda/Documents/IdeaProjects/kaggle-bliss/bug_report.py", line 21, in main
variables = my_model.init(params_key, x, False)
File "/Users/artemsereda/Documents/IdeaProjects/kaggle-bliss/bug_report.py", line 12, in __call__
x = nn.Dropout(rate=0.5, deterministic=not training)(x)
jax.errors.ConcretizationTypeError: Abstract tracer value encountered where concrete value is expected: traced array with shape bool[].
The problem arose with the `bool` function.
The error occurred while tracing the function core_fn at /Users/artemsereda/miniconda3/envs/py310/lib/python3.10/site-packages/flax/linen/transforms.py:305 for jit. This concrete value was not available in Python because it depends on the value of the argument args[2].
```
My next idea was to add the `bool` argument to `static_argnums`, as follows
```python
def main():
root_key = jax.random.PRNGKey(seed=0)
main_key, params_key, dropout_key = jax.random.split(key=root_key, num=3)
my_model = nn.jit(MyModel, static_argnums=2)(num_neurons=3)
x = jnp.empty((3, 4, 4))
variables = my_model.init(params_key, x, False)
```

This one worked, but as per the documentation, "_Calling the jitted function with different values for these constants will trigger recompilation._". The above-mentioned guide suggests using `training=True` for training steps, and `training=False` for validation steps, which will mean, I will have to re-compile full model twice in each training epoch.

Is there any way to address this?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.