apache / apache/beam

Remove the need to patch up the coder for DataflowRunner iterable side inputs on the unified worker

Open
#20,043 0 comments 0 reactions 0 assignees View on GitHub
dataflow improvement P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

FnApiRunnerTest.test_flattened_side_input failed without the fixup because the proto and JSON descriptions differed. One was "keyed" and the other was not causing failures during transcoding from the materialized form to the Fn API compatible form.

Imported from Jira [BEAM-9173](https://issues.apache.org/jira/browse/BEAM-9173). Original Jira may contain additional context.
Reported by: lcwik.

Contributor guide

Open the contributing guide

Research direction

Start with FnApiRunnerTest.test_flattened_side_input and reproduce the failure without the coder fixup. Trace how the materialized form is transcoded into the Fn API-compatible form, comparing the proto and JSON descriptions, and consider the issue done when the test passes without the patch-up and both descriptions agree about whether the input is keyed.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.