apache / apache/beam

Prohibit stacked GBKs with accumulating mode

Open
#18,684 0 comments 0 reactions 0 assignees View on GitHub
bug core java P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

The following test https://github.com/apache/beam/pull/4239 demonstrates that stacked GBKs with accumulating mode are unsafe, the same way that stacked GBKs with merging windows are unsafe.

In particular, in the pipeline: input -\> (gbk onto N keys) -\> ungroup -\> (gbk onto 1 key) -\> ungroup, e.g. suppose the first gbk receives "a" and then "b"; it will emit "a" and then "a","b" - then the second gbk will emit "a" and then "a","a","b" which is meaningless. With combine instead of GBK, it leads to double-counting.

There are cases where accumulation propagated through stacked aggregation can be desirable, but having it propagate by default is definitely the wrong thing to do. Silently changing it to discarding is likely also the wrong thing to do. So, we should reset the windowing strategy and force the user to specify accumulating mode explicitly if they would like to.

All pipelines using this currently are computing meaningless results, so rejecting them should not be considered a breaking change. However, we should still find out whether there are a lot of such pipelines or not.

Imported from Jira [BEAM-3353](https://issues.apache.org/jira/browse/BEAM-3353). Original Jira may contain additional context.
Reported by: jkff.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the behavior demonstrated in pull request 4239 and the stacked GBK pipeline described here, then trace how windowing and accumulation settings propagate across grouped operations. Done means stacked GBKs no longer silently propagate accumulating mode and pipelines must explicitly request it, with tests covering the unsafe case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.