apache / apache/beam

Outputting to an undeclared side output TupleTag should throw

Open
#18,289 0 comments 0 reactions 0 assignees View on GitHub
bug dataflow P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Currently some runners (perhaps all of them?..), when you output to a side output tag that wasn't passed to ParDo.withOutputTags, simply drop the data on the floor (or increase some aggregators).

Silent data dropping is never okay, and it's never okay to output to an undeclared tag. So instead this should throw.

Imported from Jira [BEAM-1979](https://issues.apache.org/jira/browse/BEAM-1979). Original Jira may contain additional context.
Reported by: jkff.

Contributor guide

Open the contributing guide

Research direction

Start by tracing ParDo.withOutputTags and the runner behavior when a TupleTag is not declared. Compare how the affected runners handle such output and look for existing tests around side outputs. Done means undeclared side-output data is rejected with an exception rather than silently dropped, with coverage for the relevant runner behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.