google / google/flax

Issues checkpointing optimizer state using Optax, nnx.Optimizer, and Orbax

Open
#4,423 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
7.3k
Forks
833
Avg merge
5h 11m
Merged PRs (30d)
5

Description

I am running into an error while trying to checkpoint an Optax optimizer state, wrapped as an `nnx.Optimizer`, using the Orbax checkpointing library.

```
ValueError: Unsupported type: for key: ('0', 'count'). Supported types are (, , , ).
```

I am using packages:
- flax 0.10.2
- jax 0.4.36
- jax-cuda12-pjrt 0.4.36
- jax-cuda12-plugin 0.4.36
- jaxlib 0.4.36
- optax 0.2.4
- orbax-checkpoint 0.10.2

Minimal repro:
```
from flax import nnx
import numpy as np
import orbax.checkpoint as ocp
import optax
import os
import pathlib

class MyModel(nnx.Module):
def __init__(self, rngs):
self.linear = nnx.Linear(in_features=4, out_features=1, rngs=rngs)

def __call__(self, x):
return self.linear(x)

rngs = nnx.Rngs(0)
model = MyModel(rngs)

tx = optax.adam(1e-3)
optimizer = nnx.Optimizer(model, tx)

checkpointDir = pathlib.Path('/tmp/my-checkpoints/')
checkpointer = ocp.StandardCheckpointer()

checkpointer.save(checkpointDir, optimizer.opt_state, force=True)
```

I see yall have documentation about using Orbax for model checkpointing, but don't see any official info about optimizer state checkpointing.

I see an older github issue where the Optax folks recommended `cloudpickle`: https://github.com/google-deepmind/optax/discussions/180

I tried adding a custom serialization/deserialization for `nnx.training.optimizer.OptArray` via https://orbax.readthedocs.io/en/latest/guides/checkpoint/custom_handlers.html#custom-serialization-deserialization, but then just ran into the next error:

```
ValueError: TypeHandler lookup failed for: type=
```

Maybe I'd need to also add a type handler for `OptVariable` as well as maybe even `Variable`? That seems quite annoying. Am I missing something?

Thanks for your time!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.