DropInputs on BigQueryIO.writeTableRows() sink
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
After upgrade to version 2.1.0 there are "DropInput##" transforms on every output of my pipeline, outside of their corresponding transforms. They're getting added to my
`BigQueryIO.writeTableRows()`
sinks, even though looking at the source code it appears that these "DropInput" transforms are supposed to pull from reads that do not have any outputs, so I'm not sure why they're getting added to sinks.
These transforms are flooding my graph and filling it with junk, for example: !https://i.imgur.com/jXtn9Gr.png|thumbnail!
Imported from Jira [BEAM-3017](https://issues.apache.org/jira/browse/BEAM-3017). Original Jira may contain additional context.
Reported by: MingweiSamuel.
Contributor guide
Research direction
Start at the BigQueryIO.writeTableRows() sink and inspect how DropInput## transforms are added to pipeline outputs after the 2.1.0 upgrade. Reproduce the graph described in the issue, then confirm that the sink no longer receives unrelated DropInput transforms and that the output graph is free of the reported junk.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100