apache / apache/beam

Swap to using CoGBK as grouping primitive instead of GBK

Open
#18,032 0 comments 1 reaction 0 assignees View on GitHub
backward-incompatible beam-model improvement P3 portability
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

The intent is for the semantics of both GBK and CoGBK to be
unchanged, just swapping their status as primitives.

CoGBK is a more powerful operator then GBK allowing for two key benefits:

1) SDKs are simplified: transforming a CoGBK into a GBK is trivial while the reverse is not.
2) It will be easier for runners to provide more efficient implementations of CoGBK as they will be responsible for the logic which takes their own internal grouping implementation and maps it onto a CoGBK.

This requires the following modifications to the Beam code base:

1) Make GBK a composite transform in terms of CoGBK.
2) Move the CoGBK from contrib to runners-core as an adapter*. Runners that more naturally support GBK can just use this and everything executes exactly as before.

*just like GroupByKeyViaGroupByKeyOnly and UnboundedReadFromBoundedSource

Imported from Jira [BEAM-490](https://issues.apache.org/jira/browse/BEAM-490). Original Jira may contain additional context.
Reported by: lcwik.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing GBK and CoGBK implementations, including the contrib and runners-core locations named in the issue. Verify how GBK can become a composite of CoGBK, how the adapter preserves both transforms' semantics, and that runners can continue using the existing grouping behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.