holoviz / holoviz/datashader

canvas.raster mean aggregation on integer arrays causes artifacts

Open
#655 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
3.6k
Forks
376
Avg merge
4h 32m
Merged PRs (30d)
1

Description

The regridding support in datashader is based on numba code which expects float arrays. Therefore when a user passes integer arrays to ``canvas.raster`` they are first cast to floats, then regridded and finally cast back to integers. Unfortunately this introduces highly undesirable artifacts which are very evident when using 'mean' aggregation. This is because numerical precision issues will cause values to fluctuate slightly around the integer values even in regions which are uniform in value, and when casting the floats with minor numerical precision differences to integers the minor differences are potentially amplified to a whole integer difference.

The question then is how we should handle mean/median/var/std aggregation on an integer array. I see a number of options:

1) 'mean'/'var'/'median'/'std' aggregation on an integer arrays returns floats
2) ``canvas.raster`` always returns float arrays
3) After 'mean' aggregation the float array is rounded before casting back to integers

My inclination is to go with option 1) but would welcome other suggestions.

Cc: @mmccarty

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.