cytomining / cytomining/CytoTable
Reduce join operation memory consumption
Open
enhancement
- Dominant language
- Python
- Stars
- 21
- Forks
- 6
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 6
Description
After merging #220 pre-join operations appear to require far less memory. Join operations are also reduced but could be further enhanced through SQL-based filtering (perhaps through where clauses in sub-queries), custom page selection (reading limited number of intermediate Parquet files), both those methods, or something else to limit the overall memory consumption for joins.
Contributor guide
Assessment
This issue has not been assessed yet.