UNION ALL should be implemented via Flatten
Open
dsl
improvement
P3
sql
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Currently, UNION ALL is implemented as an abstracted set operation that goes through CoGroupByKey, when a flatten will do. This is also incompatible with triggers that fire more than once.
Imported from Jira [BEAM-3721](https://issues.apache.org/jira/browse/BEAM-3721). Original Jira may contain additional context.
Reported by: kenn.
Contributor guide
Research direction
No files or tests are named. Start by tracing Beam SQL's UNION ALL implementation and its CoGroupByKey path, then inspect the corresponding Flatten handling and trigger behavior. Done means UNION ALL uses Flatten and remains correct when triggers fire more than once.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- data-engineering, stream-processing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100