dask / dask/dask-image

dask_image imread performance issue

Open
#181 24 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
226
Forks
56
PR merge metrics
No merged PRs in 30d

Description

Dear dask_image community,

I am a new dask_image user. Maybe, due to my beginner level, I am doing something wrong, but I noticed that reading a collection of images using dask_image is much slower that using single-threaded skimage. I have installed the latest dask_image version available in pypi (dask_image version 0.4.0). In the following example, I am reading 398 images, all of them with the same dimension (64x10240, uint16). Taking into account the dimensions and numbers of images, I would expect dask_image to be slighly slower that single-threaded skimage (due to the tiny dask overhead involved in opening this small number of "tiny images"), but instead the performance of dask_image is much slower (around 24x). Then I proceed to implement the image reading function in pure-dask and the performance is much better than the one obtained with dask_image. In the following I will report the benchmarks results (all the following code-snippets load the same data successfully):

import glob
import numpy as np
import skimage.io
import dask_image.imread
from dask import delayed
import dask.array as da

---------------------------------------------------------------------------------------
### Single-threaded-skimage baseline
```python
%%time
all_images = sorted(glob.glob(f"{path_images}/*.tif"))
array_images = np.zeros((len(all_images), 64, 10240), dtype=np.uint16)
for idx, image in enumerate(all_images):
array_images[idx] = skimage.io.imread(image)
```
Elapsed time: 510 milliseconds

---------------------------------------------------------------------------------------
### Using dask_image
```python
%%time
using_dask_image = dask_image.imread.imread(f"{path_images}/*.tif")
array_dask_image = using_dask_image.compute()
```
Elapsed time: 12.1 seconds

---------------------------------------------------------------------------------------
### Using pure-dask
```python
%%time
lazy_imread = delayed(skimage.io.imread) # lazy reader
lazy_arrays = [lazy_imread(image) for image in all_images]
dask_arrays = [
da.from_delayed(delayed_reader, shape=(64, 10240), dtype=np.uint16)
for delayed_reader in lazy_arrays
]
using_dask = da.stack(dask_arrays, axis=0).compute()
```
Elapsed time: 1.09 seconds

---------------------------------------------------------------------------------------
### Using dask-image with synchronous scheduler
```python
%%time
array_dask_image = using_dask_image.compute(scheduler="synchronous")
```
Elapsed time: 3 seconds

---------------------------------------------------------------------------------------
### Using dask-image with processes scheduler
```python
%%time
array_dask_image = using_dask_image.compute(scheduler="processes")
```
Elapsed time: 6.63 seconds

---------------------------------------------------------------------------------------
### Using dask-image with threads scheduler
```python
%%time
array_dask_image = using_dask_image.compute(scheduler="threads")
```
Elapsed time: 12 seconds

---------------------------------------------------------------------------------------

**Environment**:

- Dask-Image version: 0.4.0
- Dask version: 2021.01.0
- Python version: 3.8.3
- Operating System: Ubuntu 18.04
- Install method (conda, pip, source): pip

---------------------------------------------------------------------------------------

Thank you very much for your all help ;)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.