Large amount of time spent on determining datashape
- Dominant language
- Python
- Stars
- 3.6k
- Forks
- 376
- Avg merge
- 4h 32m
- Merged PRs (30d)
- 1
Description
When running aggregation on a large dataset a fairly large amount of time is spent in the ``dshape_from_dask`` function (somewhere in the region of 500-600ms). This is a major overhead when running the aggregation many times in a row and even dominates the actual aggregation time in many cases. Here is the output of a profiler running canvas.points on the census dataset:
```
77200 function calls (75103 primitive calls) in 1.021 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 1.021 1.021 {built-in method builtins.exec}
1 0.000 0.000 1.021 1.021 :2()
1 0.000 0.000 1.021 1.021 spaces.py:1093(__getitem__)
1 0.000 0.000 1.020 1.020 spaces.py:890(_execute_callback)
1 0.000 0.000 1.020 1.020 spaces.py:540(__call__)
1 0.000 0.000 1.020 1.020 __init__.py:430(dynamic_operation)
1 0.000 0.000 1.020 1.020 __init__.py:415(_process)
1 0.000 0.000 1.020 1.020 operation.py:137(process_element)
2/1 0.000 0.000 1.020 1.020 operation.py:113(_apply)
1 0.000 0.000 1.019 1.019 datashader.py:782(_process)
18/7 0.000 0.000 1.018 0.145 dimension.py:712(map)
1 0.000 0.000 1.017 1.017 operation.py:146(__call__)
1 0.000 0.000 0.998 0.998 datashader.py:416(_process)
1 0.000 0.000 0.964 0.964 core.py:144(points)
1 0.000 0.000 0.964 0.964 core.py:492(bypixel)
5 0.000 0.000 0.897 0.179 local.py:405(get_async)
3 0.000 0.000 0.897 0.299 threaded.py:33(get)
24 0.000 0.000 0.894 0.037 local.py:150(queue_get)
24 0.000 0.000 0.894 0.037 queue.py:147(get)
12 0.000 0.000 0.893 0.074 threading.py:263(wait)
96 0.893 0.009 0.893 0.009 {method 'acquire' of '_thread.lock' objects}
1 0.066 0.066 0.491 0.491 utils.py:368(dshape_from_dask) <-----
2/1 0.000 0.000 0.472 0.472 utils.py:51(__call__)
1 0.000 0.000 0.471 0.471 dask.py:20(dask_pipeline)
```
You can see that almost half the time is spent in the ``dshape_from_dask`` function, and as you aggregate a smaller region this time dominates even more.
It would be great if this could be sped up in general but it might be sufficient to allow passing in the datashape in cases where you are doing a lot of aggregation on the same datasource (e.g. when generating a tileset) and want to avoid this 0.5 second penalty each time.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.