google / google/orbax

Batch per-slice device_put in transfer_arrays_to_host to reduce D2H dispatch overhead

Open
#3,493 2 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Python
Stars
535
Forks
101
Avg merge
1d 13h
Merged PRs (30d)
23

Description

### Summary

In `replica_slices.transfer_arrays_to_host`, the pinned-host transfer path issues one `jax.device_put` per replica slice ([replica_slices.py#L451](https://github.com/google/orbax/blob/9bd1c2707fa97f01ca478a1e293df6acfec4c7dc/checkpoint/orbax/checkpoint/_src/serialization/replica_slices.py#L451)). Each `device_put` carries a fixed per-call dispatch cost independent of slice size, so when saving many arrays the dispatch count can dominate the transfer step.

### Proposal

`jax.device_put` accepts a list of arrays plus a matching list of shardings and issues them in a single dispatch. Batching all pinned-host slices into one `device_put` pays the dispatch cost once. Non-pinned slices keep the existing `copy_to_host_async()` path. This is behavior-preserving: the same slices/data are transferred with the same replica-parallel semantics, and peak host memory is unchanged (all buffers were already held until the final await). e.g.:

```python
# Current: one device_put per slice
for rslice in rslices:
data = rslice.data()
if use_pinned_host_transfer(data.device):
data = jax.device_put(
data,
jax.sharding.SingleDeviceSharding(data.device, memory_kind='pinned_host'),
)
else:
data.copy_to_host_async()

# Proposed: batch all pinned slices into a single device_put
pinned = [s.data() for s in rslices if use_pinned_host_transfer(s.data().device)]
on_host = jax.device_put(
pinned,
[jax.sharding.SingleDeviceSharding(d.device, memory_kind='pinned_host') for d in pinned],
)
```

### Question for maintainers

Is a single batched `device_put` across all pinned slices acceptable, or was the per-slice form chosen for a specific reason (ordering, memory, or backend constraints)?

Contributor guide

Open the contributing guide

Research direction

Start in checkpoint/orbax/checkpoint/_src/serialization/replica_slices.py at transfer_arrays_to_host and inspect the pinned-host and copy_to_host_async paths around line 451. Confirm the transfer remains replica-parallel, non-pinned slices retain their existing path, and pinned slices use one dispatch without increasing peak host memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
66/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.