dask / dask/dask-image

Performance issue with `sum`

Open
#238 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
226
Forks
56
PR merge metrics
No merged PRs in 30d

Description

Working on a modest-size cube, I found that `scipy.ndimage.sum` is ~100-300x faster than `dask_image.ndmeasure.sum_labels`

```python
import numpy as np, scipy.ndimage
blah = np.random.randn(19,512,512)
msk = blah > 3
lab, ct = scipy.ndimage.label(msk)
%timeit scipy.ndimage.sum(msk, labels=lab, index=range(1, ct+1))
# 117 ms ± 2.85 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

vs

```python
rslt = ndmeasure.sum_labels(msk, label_image=lab, index=range(1, ct+1))
rslt
# dask.array
rslt.compute()
# [########################################] | 100% Completed | 22.9s
```

Note also that the task creation takes nontrivial time:
```python
%timeit ndmeasure.sum_labels(msk, label_image=lab, index=range(1, ct+1))
# 15.4 s ± 2.02 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

While I understand that there ought to be some cost to running this processing through a graph with dask, this seems excessively slow. Is there a different approach I should be taking, or is this a bug?

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported benchmarks for scipy.ndimage.sum and ndmeasure.sum_labels using the 19×512×512 example and the displayed label range. Inspect the sum_labels entry point and its Dask task creation, including the cost before compute. Done means establishing whether the overhead is expected or a performance bug and recording the relevant evidence or next step.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.