datashader memory consumption on very large dataset
- Dominant language
- Python
- Stars
- 3.6k
- Forks
- 376
- Avg merge
- 4h 32m
- Merged PRs (30d)
- 1
Description
We are dealing with total of 140 million points from ~500 files. It is great that datashader can draw it out, but it also took 100GB of peak memory and about 40 minutes on a couple of cores getting the data into dataframe while the overall system cpu utilization is pretty spare. Is there anyway to both reduce the memory requirement and increase/parallelize the CPU usage?
Thanks
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files, tests, or entry points. Start by reproducing the reported 140-million-point workload from roughly 500 files, then measure peak memory, elapsed time, and CPU utilization; done would require a concrete way to reduce memory use and improve or parallelize CPU usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-visualization, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100