holoviz / holoviz/datashader

_colorize can exceed available memory if there is a large number of unique categories

Open
#907 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.6k
Forks
376
Avg merge
4h 32m
Merged PRs (30d)
1

Description

The pipeline description says the following:

```
...only one stage (Aggregation) requires access to the entire dataset. The remaining stages use a fixed-sized data structure regardless of the input dataset...
```

However, I have observed that if I have a dataset with a **very large number** of unique categories (for this example let's say > 50,000) then colorize will allocate a very large array.

```
File "datashader/transfer_functions/__init__.py", line 352, in _colorize
r = (data.dot(rs)/total).astype(np.uint8)
MemoryError: Unable to allocate 27.2 GiB for an array with shape (256, 256, 55804) and data type int64
```

It appears that the `_colorize` datastructure is only fixed in the image size dimensions and scales O(n) in the category dimension.

#### ALL software version info
Datashader 0.10.0

#### Description of expected behavior and the observed behavior
Ideal behavior is that _colorize could handle an arbitrary number of categories, similar to other aspects of Datashader. Improving the memory utilization of colorize would be second best. Documentation of limitation and/or work around is also be fine.

#### Complete, minimal, self-contained example code that reproduces the issue

Will provide in follow-up. Thanks as always.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.