Use runtime information to improve CoGroupByKey caching
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Currently, CoGroupByKey creates UnionTables that are Flattened. The Flattened output is processed by a GroupByKey to produce a CoGbkResult (via ConstructCoGbkResultFn).
Given the performance of CoGBK is greatly impacted based on the which elements are cached in the (finitely sized) in-memory results, it would be useful if CoGbkResult can use runtime information to prioritize which elements are stored in-memory.
Imported from Jira [BEAM-13541](https://issues.apache.org/jira/browse/BEAM-13541). Original Jira may contain additional context.
Reported by: pedapudi.
Contributor guide
Research direction
Start by tracing CoGroupByKey through UnionTables, Flattened, GroupByKey, and ConstructCoGbkResultFn, then review the imported BEAM-13541 Jira issue for context. The work is complete when CoGbkResult uses runtime information to prioritize in-memory elements, with the caching behavior and performance impact validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100