google / google/flax

Flax NNX and Orbax Checkpointing require hacking to work together

Open
#4,383 6 comments 9 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
7.3k
Forks
833
Avg merge
5h 11m
Merged PRs (30d)
5

Description

I'm building a system using `flax.nnx` and `orbax.checkpointing`. However, it is overly complicated on how to save and restore models due to the new `jax.random.key()` being used in `flax.nnx` rather than `jax.random.PRNGkey()`.

I have had to create a workaround where all layers with `rng` and `key` in their path are changed from `dtype=key` to a format appropriate for saving. Then, upon restoration, they need to be shanged back.

I am attaching a link to a notebook explaining what I've done but I would be keen to hear if there are simpler workarounds? Or, preferably, if there is a way to simple save and restore models?

https://colab.research.google.com/drive/1ozln9ejG7eRtxvbkqHYU3K6OyPvveH9w?usp=sharing

Note: I am also adding an issue to orbax to see if there is a fix their side ([#1337](https://github.com/google/orbax/issues/1337)).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.