UNION should be implemented via flatten & distinct
Open
dsl
improvement
P3
sql
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Currently, UNION is implemented as a set operator via CoGroupByKey when a flatten **** distinct will suffice. This also will be compatible with more types of triggers.
Imported from Jira [BEAM-3722](https://issues.apache.org/jira/browse/BEAM-3722). Original Jira may contain additional context.
Reported by: kenn.
Contributor guide
Research direction
Start by locating the UNION implementation and the CoGroupByKey path in Apache Beam, then inspect how flatten and distinct are used elsewhere. Confirm the relevant trigger behavior and existing tests before changing the implementation; done means UNION uses the simpler path without losing distinct set semantics or trigger compatibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data, stream-processing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100