dask / dask/dask-image

Loading stack of images slow

Open
#121 13 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
226
Forks
56
PR merge metrics
No merged PRs in 30d

Description

I have about 20k 500px500p images stored in a directory.

My ultimate goal is to reshape them into only a few images, blocking the 500s into the few larger images.

I first wanted to see how fast I could load the images.
```
arr=dask_image.imread.imread('*.png')[:100].compute()
```
versus
```
from PIL import Image
arr=[Image.open(img) for img in glob.glob('*.png')[:100]]
```
The second task is performing much much much faster than the first task (first task hangs).

Can you help me understand why this is so and what I can do to speed this up?

Chunk size is (1,500,500,3) but I changed it to (25,500,500,3) and it didn't really make a difference. I would normally expect this to be a fast and easy Dask operation. I even started processing and threaded distributed clients with no luck.

Help is very much appreciated. Thanks!

Contributor guide

Open the contributing guide

Research direction

No repository files or tests are named. Start by reproducing the dask_image.imread.imread('*.png')[:100].compute() and PIL comparison, then inspect the effects of the reported chunk sizes and threaded or distributed clients. Done means explaining the slowdown and identifying a validated way to make this image-loading workflow faster.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.