google / google/flax

Why does LSTMCell keep rngs in its state?

Open
#4,509 3 comments 3 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
7.3k
Forks
833
Avg merge
5h 11m
Merged PRs (30d)
5

Description

It seems that LSTMCell keeps rngs in its state:
https://github.com/google/flax/blob/a8a192ff167f8b25b9b568cfece44ef043a82dad/flax/nnx/nn/recurrent.py#L137

Is this intentional? Why?

I stumbled upon this because my recipe for checkpointing breaks when my model contains an LSTM:

```
import orbax.checkpoint as ocp
def savemodel(model, path):
_, state = nnx.split(model)
checkpointer = ocp.StandardCheckpointer()
checkpointer.save(path, state)
```

Calling `savemodel(model, path)` throws:
```
TypeError: JAX array with PRNGKey dtype cannot be converted to a NumPy array. Use jax.random.key_data(arr) if you wish to extract the underlying integer array.
```

This was surprising because I've been using that recipe before and never had a problem while using other non-LSTM modules.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.