google / google/flax

Consider deprecating common_utils.shard

Open
#948 2 comments 3 reactions 1 assignee Claimed by @jheek View on GitHub
Priority: P2 - no schedule
Dominant language
Jupyter Notebook
Stars
7.3k
Forks
833
Avg merge
5h 11m
Merged PRs (30d)
5

Description

The following pattern is common is user code and our examples:
```
def shard(pytree, n_devices):
def _shard_array(array):
return array.reshape((n_devices, -1) + array.shape[1:])
return jax.tree_map(_shard_array, pytree)
```

The shard utility is part of `flax.common_utils`.

This pattern has 2 issues:
1. It complicates the training loop because part of the preprocessing is done async in tf.data while shard operates in the training loop
2. The code works both on NumPy and JAX arrays but has completely different performance characteristics. On numpy this is a view changes which is practically free. Whereas on Jax arrays this will trigger a copy that runs on the device which can cost a lot of time and memory in particular on TPUs.

**Proposed solution**

Make sure input data already has the correct shape at the end of the input pipeline. In tf.data we can do this by using a double batch call:
```
dataset = dataset.batch(device_batch_size).batch(num_devices)
```

The advantage here is that the input pipeline always provides use with correctly shaped data. No reshaping is done in the critical loop and there is no confusion about numpy vs jax arrays.

TODO:
- [ ] Deprecate flax.common_utils.shard
- [ ] Actively recommend using the alternative pattern
- [ ] Correct the flax examples WMT example, nlp_seq and pixelcnn

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.