Random access array take using P2P
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Slicing in dask array effectively generates a task per contiguous subslice per chunk.
For the worst case of random indexing this generates a slice/task for every row along this dimension. Dask is currently raising a `PerformanceWarning` once we detect this situation, see https://github.com/dask/dask/blob/b4b33caed8fc9cf77c9332442ab11cf00f90bb42/dask/array/slicing.py#L630-L641
Worst case example
```python
import dask.array as da
import numpy as np
x = da.random.random((10, 20), chunks=(10, 10))
idx = np.random.randint(0, x.shape[1], x.shape[1])
y = x[:, idx]
```
This random access pattern is another shuffle pattern and we should be able to offer an efficient solution to this using our P2P infrastructure
see also https://github.com/pydata/xarray/issues/9220
Contributor guide
Assessment
This issue has not been assessed yet.