holoviz / holoviz/datashader

Xarray + DaskArray out-of-memory on QuadMesh where raster works

Open
#972 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.6k
Forks
376
Avg merge
4h 32m
Merged PRs (30d)
1

Description

My Python kernel gets killed because of an out-of-memory issue when generating a `quadmesh` from a large Xarray DataArray using DaskArrays as data. Visualising the same DataArray with `raster` works using Dask's out-of-core computation support.

When looking at the Dask scheduler I noticed that nothing gets scheduled when creating the `quadmesh` (I think because the whole DataArray is forced into memory). While using `raster` shows a nice computation graph and is able to compute the figure out-of-core without any issues.

To illustrate this I created two notebooks trying to render a figure from the same DataArray

1. Successfully running `raster`: https://gist.github.com/peterroelants/0624834713a1388c7f57d3cafd9b800b
- Dask schedules the computations needed to compute the image (See screenshots below).
2. Failure of running `quadmesh`: https://gist.github.com/peterroelants/dd5375ed5d58e1dfd72bc2003539124d
- Dask scheduler stays empty when running this. No tasks are scheduled. Presumable because the whole DataArray is forced into memory before any of the computation happens.

#### ALL software version info

```
Python implementation: CPython
Python version : 3.8.6
IPython version : 7.19.0

Compiler : GCC 7.5.0
OS : Linux
Release : 5.4.0-54-generic
Machine : x86_64
Processor : x86_64
CPU cores : 4
Architecture: 64bit

sys : 3.8.6 | packaged by conda-forge | (default, Oct 7 2020, 19:08:05)
[GCC 7.5.0]
holoviews : 1.13.5
datashader: 0.11.2a5
xarray : 0.16.1
numpy : 1.19.4
dask : 2.30.0
```

`datashader` build from `master` with commit-id `fd938888feca3a42bdfb42462d098f758a954dd8`

#### Description of expected behavior and the observed behavior

I would expect that `quadmesh` does not try to load the whole DataArray in memory and tries to leverage Dask's out-of-core computation infrastructure, similar to how `raster` does this. I would expect from the documentation at https://datashader.org/user_guide/Performance.html that Xarray + DaskArray is supported.

#### Complete, minimal, self-contained example code that reproduces the issue

See notebook at: https://gist.github.com/peterroelants/dd5375ed5d58e1dfd72bc2003539124d

#### Stack traceback and/or browser JavaScript console output

```
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk. Perhaps some other process is leaking memory? Process memory: 13.04 GB -- Worker memory limit: 10.00 GB
```

Before Python kernel gets killed because it runs out of memory.

#### Screenshots of rasterize leveraging Dask's out-of-core:

Screenshots taken when successfully running `raster`: https://gist.github.com/peterroelants/0624834713a1388c7f57d3cafd9b800b
- Notice the Dask scheduler nicely scheduling the `rasterize` tasks (This does not happen with `quadmesh`): ![dask_raster_02](https://user-images.githubusercontent.com/951093/100599221-07465b00-3300-11eb-9cdb-0a1a215478ee.png)
- Notice the `rasterize` task graph in the Dask scheduler (This stays at "Scheduler is empty" when running `quadmesh`): ![Dask_raster_graph_02](https://user-images.githubusercontent.com/951093/100599258-14634a00-3300-11eb-9955-a4291e155ac6.png)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the `quadmesh` and `raster` entry points and compare their behavior using the two linked notebooks. Reproduce the out-of-memory case with the quadmesh notebook, then verify that equivalent Xarray DataArray workloads schedule Dask tasks and complete without loading the entire array into memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-visualization, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.