Remove workaround for BQ transform for Dataflow
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Dataflow incorrectly uses the Flatten input PCollection coder when it performs an optimization instead of the output PCollection coder which can lead to issues if these coders differ.
The workaround was introduced in [https://github.com/apache/beam/pull/11103](https://github.com/apache/beam/pull/11103)
Imported from Jira [BEAM-9494](https://issues.apache.org/jira/browse/BEAM-9494). Original Jira may contain additional context.
Reported by: lcwik.
Contributor guide
Research direction
Start by reviewing the workaround introduced in Apache Beam PR 11103 and the issue's description of Dataflow's Flatten coder optimization. Trace the relevant Beam/Dataflow entry points to determine where the workaround is applied; done means it can be removed without Dataflow using the input coder when the output coder differs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100