google-deepmind / google-deepmind/optax
optax.MultiSteps out of memory
Open
type:bug
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 369
- Avg merge
- 10h 15m
- Merged PRs (30d)
- 7
Description
I always get an out of memory error using optax.MultiSteps, even when every_k_schedule=1.
Using optax.apply_every(k=1) in a chain works fine.
```
optimizer = optax.chain(
optax.clip_by_global_norm(0.5),
optax.adam(lr),
#optax.apply_every(k=1)
)
optimizer = optax.MultiSteps(optimizer, every_k_schedule=1)
```
Later I'm using
`opt_state = optimizer.init(params)`
and
```
updates, opt_state = optimizer.update(grads, opt_state, params)
params = optax.apply_updates(params, updates)
```
I have no idea what I could be doing wrong. I'm not changing anything else, like batch size.
Contributor guide
Assessment
This issue has not been assessed yet.