dask / dask/distributed

Random access array take using P2P

Open
#8,774 2 comments 0 reactions 0 assignees View on GitHub
feature shuffle
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Slicing in dask array effectively generates a task per contiguous subslice per chunk.

For the worst case of random indexing this generates a slice/task for every row along this dimension. Dask is currently raising a `PerformanceWarning` once we detect this situation, see https://github.com/dask/dask/blob/b4b33caed8fc9cf77c9332442ab11cf00f90bb42/dask/array/slicing.py#L630-L641

Worst case example

```python
import dask.array as da
import numpy as np

x = da.random.random((10, 20), chunks=(10, 10))
idx = np.random.randint(0, x.shape[1], x.shape[1])

y = x[:, idx]
```

This random access pattern is another shuffle pattern and we should be able to offer an efficient solution to this using our P2P infrastructure

see also https://github.com/pydata/xarray/issues/9220

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.