cytomining / cytomining/CytoTable

Reduce join operation memory consumption

Open
#223 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
21
Forks
6
Avg merge
2d 3h
Merged PRs (30d)
6

Description

After merging #220 pre-join operations appear to require far less memory. Join operations are also reduced but could be further enhanced through SQL-based filtering (perhaps through where clauses in sub-queries), custom page selection (reading limited number of intermediate Parquet files), both those methods, or something else to limit the overall memory consumption for joins.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.