Xarray + DaskArray out-of-memory on QuadMesh where raster works
- Dominant language
- Python
- Stars
- 3.6k
- Forks
- 376
- Avg merge
- 4h 32m
- Merged PRs (30d)
- 1
Description
My Python kernel gets killed because of an out-of-memory issue when generating a `quadmesh` from a large Xarray DataArray using DaskArrays as data. Visualising the same DataArray with `raster` works using Dask's out-of-core computation support.
When looking at the Dask scheduler I noticed that nothing gets scheduled when creating the `quadmesh` (I think because the whole DataArray is forced into memory). While using `raster` shows a nice computation graph and is able to compute the figure out-of-core without any issues.
To illustrate this I created two notebooks trying to render a figure from the same DataArray
1. Successfully running `raster`: https://gist.github.com/peterroelants/0624834713a1388c7f57d3cafd9b800b
- Dask schedules the computations needed to compute the image (See screenshots below).
2. Failure of running `quadmesh`: https://gist.github.com/peterroelants/dd5375ed5d58e1dfd72bc2003539124d
- Dask scheduler stays empty when running this. No tasks are scheduled. Presumable because the whole DataArray is forced into memory before any of the computation happens.
#### ALL software version info
```
Python implementation: CPython
Python version : 3.8.6
IPython version : 7.19.0
Compiler : GCC 7.5.0
OS : Linux
Release : 5.4.0-54-generic
Machine : x86_64
Processor : x86_64
CPU cores : 4
Architecture: 64bit
sys : 3.8.6 | packaged by conda-forge | (default, Oct 7 2020, 19:08:05)
[GCC 7.5.0]
holoviews : 1.13.5
datashader: 0.11.2a5
xarray : 0.16.1
numpy : 1.19.4
dask : 2.30.0
```
`datashader` build from `master` with commit-id `fd938888feca3a42bdfb42462d098f758a954dd8`
#### Description of expected behavior and the observed behavior
I would expect that `quadmesh` does not try to load the whole DataArray in memory and tries to leverage Dask's out-of-core computation infrastructure, similar to how `raster` does this. I would expect from the documentation at https://datashader.org/user_guide/Performance.html that Xarray + DaskArray is supported.
#### Complete, minimal, self-contained example code that reproduces the issue
See notebook at: https://gist.github.com/peterroelants/dd5375ed5d58e1dfd72bc2003539124d
#### Stack traceback and/or browser JavaScript console output
```
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk. Perhaps some other process is leaking memory? Process memory: 13.04 GB -- Worker memory limit: 10.00 GB
```
Before Python kernel gets killed because it runs out of memory.
#### Screenshots of rasterize leveraging Dask's out-of-core:
Screenshots taken when successfully running `raster`: https://gist.github.com/peterroelants/0624834713a1388c7f57d3cafd9b800b
- Notice the Dask scheduler nicely scheduling the `rasterize` tasks (This does not happen with `quadmesh`): 
- Notice the `rasterize` task graph in the Dask scheduler (This stays at "Scheduler is empty" when running `quadmesh`): 
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the `quadmesh` and `raster` entry points and compare their behavior using the two linked notebooks. Reproduce the out-of-memory case with the quadmesh notebook, then verify that equivalent Xarray DataArray workloads schedule Dask tasks and complete without loading the entire array into memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-visualization, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100