holoviz / holoviz/datashader

datashader memory consumption on very large dataset

Open
#638 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
3.6k
Forks
376
Avg merge
4h 32m
Merged PRs (30d)
1

Description

We are dealing with total of 140 million points from ~500 files. It is great that datashader can draw it out, but it also took 100GB of peak memory and about 40 minutes on a couple of cores getting the data into dataframe while the overall system cpu utilization is pretty spare. Is there anyway to both reduce the memory requirement and increase/parallelize the CPU usage?

Thanks

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points. Start by reproducing the reported 140-million-point workload from roughly 500 files, then measure peak memory, elapsed time, and CPU utilization; done would require a concrete way to reduce memory use and improve or parallelize CPU usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-visualization, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.