google / google/flax

nn.remat_scan doesn't work with nn.with_partitioning

Open
#3,870 1 comment 0 reactions 0 assignees View on GitHub
Priority: P2 - no schedule
Dominant language
Jupyter Notebook
Stars
7.3k
Forks
833
Avg merge
5h 11m
Merged PRs (30d)
5

Description

### Problem you have encountered:

Using `nn.remat_scan` on a layer that defines variables using `nn.with_partitioning` fails.

### What you expected to happen:

It seems like the default behavior should be to replicate along the scan axis if nothing else is specified.

While I've seen the comments about preferring scan(remat(...)) (https://flax.readthedocs.io/en/latest/faq.html#is-flax-linen-remat-scan-the-same-as-scan-remat), remat_scan is very useful because it provides the right defaults for repeatedly applying layers whose input and output is the same shape, even if the behavior regarding remat is a bit confusing.

### Logs, error messages, etc:

`PartitioningUnspecifiedError: Trying to transform a Partitioned variable but "partition_name" is not specified in metadata_params`

### Steps to reproduce:

```python
layer = nn.remat_scan(nn.Dense, lengths=(2,))(
3, kernel_init=nn.with_partitioning(nn.initializers.lecun_normal(), ('replica',))
)

layer.init(jax.random.key(0), jnp.zeros((3,)))
```

https://colab.research.google.com/drive/1f7f4HdNsROOGXzkWQIriAp5CocGwxQ1y#scrollTo=7P0lwCb9aSLp

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.