_colorize can exceed available memory if there is a large number of unique categories
- Dominant language
- Python
- Stars
- 3.6k
- Forks
- 376
- Avg merge
- 4h 32m
- Merged PRs (30d)
- 1
Description
The pipeline description says the following:
```
...only one stage (Aggregation) requires access to the entire dataset. The remaining stages use a fixed-sized data structure regardless of the input dataset...
```
However, I have observed that if I have a dataset with a **very large number** of unique categories (for this example let's say > 50,000) then colorize will allocate a very large array.
```
File "datashader/transfer_functions/__init__.py", line 352, in _colorize
r = (data.dot(rs)/total).astype(np.uint8)
MemoryError: Unable to allocate 27.2 GiB for an array with shape (256, 256, 55804) and data type int64
```
It appears that the `_colorize` datastructure is only fixed in the image size dimensions and scales O(n) in the category dimension.
#### ALL software version info
Datashader 0.10.0
#### Description of expected behavior and the observed behavior
Ideal behavior is that _colorize could handle an arbitrary number of categories, similar to other aspects of Datashader. Improving the memory utilization of colorize would be second best. Documentation of limitation and/or work around is also be fine.
#### Complete, minimal, self-contained example code that reproduces the issue
Will provide in follow-up. Thanks as always.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.