holoviz / holoviz/datashader

Large amount of time spent on determining datashape

Open
#633 33 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.6k
Forks
376
Avg merge
4h 32m
Merged PRs (30d)
1

Description

When running aggregation on a large dataset a fairly large amount of time is spent in the ``dshape_from_dask`` function (somewhere in the region of 500-600ms). This is a major overhead when running the aggregation many times in a row and even dominates the actual aggregation time in many cases. Here is the output of a profiler running canvas.points on the census dataset:

```
77200 function calls (75103 primitive calls) in 1.021 seconds

Ordered by: cumulative time

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 1.021 1.021 {built-in method builtins.exec}
1 0.000 0.000 1.021 1.021 :2()
1 0.000 0.000 1.021 1.021 spaces.py:1093(__getitem__)
1 0.000 0.000 1.020 1.020 spaces.py:890(_execute_callback)
1 0.000 0.000 1.020 1.020 spaces.py:540(__call__)
1 0.000 0.000 1.020 1.020 __init__.py:430(dynamic_operation)
1 0.000 0.000 1.020 1.020 __init__.py:415(_process)
1 0.000 0.000 1.020 1.020 operation.py:137(process_element)
2/1 0.000 0.000 1.020 1.020 operation.py:113(_apply)
1 0.000 0.000 1.019 1.019 datashader.py:782(_process)
18/7 0.000 0.000 1.018 0.145 dimension.py:712(map)
1 0.000 0.000 1.017 1.017 operation.py:146(__call__)
1 0.000 0.000 0.998 0.998 datashader.py:416(_process)
1 0.000 0.000 0.964 0.964 core.py:144(points)
1 0.000 0.000 0.964 0.964 core.py:492(bypixel)
5 0.000 0.000 0.897 0.179 local.py:405(get_async)
3 0.000 0.000 0.897 0.299 threaded.py:33(get)
24 0.000 0.000 0.894 0.037 local.py:150(queue_get)
24 0.000 0.000 0.894 0.037 queue.py:147(get)
12 0.000 0.000 0.893 0.074 threading.py:263(wait)
96 0.893 0.009 0.893 0.009 {method 'acquire' of '_thread.lock' objects}
1 0.066 0.066 0.491 0.491 utils.py:368(dshape_from_dask) <-----
2/1 0.000 0.000 0.472 0.472 utils.py:51(__call__)
1 0.000 0.000 0.471 0.471 dask.py:20(dask_pipeline)
```

You can see that almost half the time is spent in the ``dshape_from_dask`` function, and as you aggregate a smaller region this time dominates even more.

It would be great if this could be sped up in general but it might be sufficient to allow passing in the datashape in cases where you are doing a lot of aggregation on the same datasource (e.g. when generating a tileset) and want to avoid this 0.5 second penalty each time.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.