google / google/flax

[linen] Undocumented RNN errors

Open
#4,513 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
7.3k
Forks
833
Avg merge
5h 11m
Merged PRs (30d)
5

Description

Provide as much information as possible. At least, this should include a description of your issue and steps to reproduce the problem. If possible also provide a summary of what steps or workarounds you have already tried.

### System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 22.04 WSL
- Flax, jax, jaxlib versions (obtain with `pip show flax jax jaxlib`: flax = 0.10.2 jax=0.5.0, jaxlib=0.5.0
- Python version: 12
- GPU/TPU model and memory: 1x Tesla V100, 32.0 GB
- CUDA version (if applicable):

### Problem you have encountered:
Hello Flax,
I'm stuck debugging linen.Bidirectional because it does not throw errors.
Logic tells me to do something like this:
```python
def setup(self):

self.rnn_var = nn.recurrent.Bidirectional(
nn.RNN(
nn.recurrent.OptimizedLSTMCell(self.rnn_var_lstm_cell_size),
return_carry=True,
),
nn.RNN(
nn.recurrent.OptimizedLSTMCell(self.rnn_var_lstm_cell_size),
return_carry=True,
),
return_carry=True,
) # num_layers=10,

self.initial_rnn_carry = (
( # note: don't forget to add ibatch size in the future.
nn.initializers.zeros(
jax.random.PRNGKey(0),
(self.rnn_var_lstm_cell_size,),
), # c (cell state)
nn.initializers.zeros(
jax.random.PRNGKey(0),
(self.rnn_var_lstm_cell_size,),
),
), # h (hidden state)
(
nn.initializers.zeros(
jax.random.PRNGKey(0),
(self.rnn_var_lstm_cell_size,),
), # c (cell state) for backward
nn.initializers.zeros(
jax.random.PRNGKey(0),
(self.rnn_var_lstm_cell_size,),
),
), # h (hidden state) for backward
)
# ...

def __call__(self, x: dict[str, Array]) -> Array:
import pdb

pdb.set_trace()
rnn_var_carry, _processed_vars = self.rnn_var_rnn(
nodes_gather_x, initial_carry=self.initial_rnn_carry, return_carry=True
)
#failure here^
```
And the problem is, I don't get understandable error messages, at all. For example this:
```
File "/Problemologist-flax/dreamerv3_flax/encoder.py", line 159, in __call__
rnn_var_rnn_carry, _processed_rnn_var = self.rnn_var_rnn(
^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/flax/linen/recurrent.py", line 1315, in __call__
carry_forward, outputs_forward = self.forward_rnn(
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/flax/linen/recurrent.py", line 1135, in __call__
scan_output = scan(self.cell, carry, inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/flax/core/axes_scan.py", line 152, in scan_fn
_, out_pvals, _ = pe.trace_to_jaxpr_nounits(f_flat, in_pvals)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/flax/core/axes_scan.py", line 124, in body_fn
broadcast_out, c, ys = fn(broadcast_in, c, *xs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/flax/linen/recurrent.py", line 1114, in scan_fn
carry, y = cell(carry, x)
^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/flax/linen/recurrent.py", line 336, in __call__
dense_params_i[component] = DenseParams(
^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/flax/linen/recurrent.py", line 214, in __call__
(inputs.shape[-1], self.features),
~~~~~~~~~~~~^^^^ #

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.