Batch per-slice device_put in transfer_arrays_to_host to reduce D2H dispatch overhead
- Dominant language
- Python
- Stars
- 535
- Forks
- 101
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 23
Description
### Summary
In `replica_slices.transfer_arrays_to_host`, the pinned-host transfer path issues one `jax.device_put` per replica slice ([replica_slices.py#L451](https://github.com/google/orbax/blob/9bd1c2707fa97f01ca478a1e293df6acfec4c7dc/checkpoint/orbax/checkpoint/_src/serialization/replica_slices.py#L451)). Each `device_put` carries a fixed per-call dispatch cost independent of slice size, so when saving many arrays the dispatch count can dominate the transfer step.
### Proposal
`jax.device_put` accepts a list of arrays plus a matching list of shardings and issues them in a single dispatch. Batching all pinned-host slices into one `device_put` pays the dispatch cost once. Non-pinned slices keep the existing `copy_to_host_async()` path. This is behavior-preserving: the same slices/data are transferred with the same replica-parallel semantics, and peak host memory is unchanged (all buffers were already held until the final await). e.g.:
```python
# Current: one device_put per slice
for rslice in rslices:
data = rslice.data()
if use_pinned_host_transfer(data.device):
data = jax.device_put(
data,
jax.sharding.SingleDeviceSharding(data.device, memory_kind='pinned_host'),
)
else:
data.copy_to_host_async()
# Proposed: batch all pinned slices into a single device_put
pinned = [s.data() for s in rslices if use_pinned_host_transfer(s.data().device)]
on_host = jax.device_put(
pinned,
[jax.sharding.SingleDeviceSharding(d.device, memory_kind='pinned_host') for d in pinned],
)
```
### Question for maintainers
Is a single batched `device_put` across all pinned slices acceptable, or was the per-slice form chosen for a specific reason (ordering, memory, or backend constraints)?
Contributor guide
Research direction
Start in checkpoint/orbax/checkpoint/_src/serialization/replica_slices.py at transfer_arrays_to_host and inspect the pinned-host and copy_to_host_async paths around line 451. Confirm the transfer remains replica-parallel, non-pinned slices retain their existing path, and pinned slices use one dispatch without increasing peak host memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 66/100