apache / apache/beam

DropInputs on BigQueryIO.writeTableRows() sink

Open
#18,506 0 comments 0 reactions 0 assignees View on GitHub
bug dataflow P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

After upgrade to version 2.1.0 there are "DropInput##" transforms on every output of my pipeline, outside of their corresponding transforms. They're getting added to my
`BigQueryIO.writeTableRows()`
sinks, even though looking at the source code it appears that these "DropInput" transforms are supposed to pull from reads that do not have any outputs, so I'm not sure why they're getting added to sinks.

These transforms are flooding my graph and filling it with junk, for example: !https://i.imgur.com/jXtn9Gr.png|thumbnail!

Imported from Jira [BEAM-3017](https://issues.apache.org/jira/browse/BEAM-3017). Original Jira may contain additional context.
Reported by: MingweiSamuel.

Contributor guide

Open the contributing guide

Research direction

Start at the BigQueryIO.writeTableRows() sink and inspect how DropInput## transforms are added to pipeline outputs after the 2.1.0 upgrade. Reproduce the graph described in the issue, then confirm that the sink no longer receives unrelated DropInput transforms and that the output graph is free of the reported junk.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.