holoviz / holoviz/datashader

Dask dataframe chunksize/partioning affects line aggregation

Open
#464 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
3.6k
Forks
376
Avg merge
4h 32m
Merged PRs (30d)
1

Description

When using a dask dataframe during line aggregation the chunksize affects the result, this is likely because different chunks aren't processed together effectively splitting the path. In this example I create dask dataframes with varying chunksizes and visualize the effect:

```python
%%opts Image [aspect='square' xaxis=None yaxis=None sublabel_format=''] Layout [hspace=0.1 vspace=0.1]
import dask.dataframe as dd
from holoviews.operation.datashader import datashade, aggregate
paths = hv.Path([[(i+x, i*10) for i in range(10)] for x in range(10)])
hv.Layout([aggregate(hv.Curve(dd.from_pandas(paths.dframe(), chunksize=p)),
dynamic=False, width=10, height=10).relabel('Chunksize: %d' % p)
for p in range(1, 11)])
```

![image](https://user-images.githubusercontent.com/1550771/30552737-ec5e1d8a-9c96-11e7-9031-47c95d8f63a1.png)

You can see that basically every chunksize or number of partitions produces a slightly different result.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.